A pair of model cards (模型卡) lies open on the table between two seats, printed because the technical director does not trust himself to read them on a phone. Both describe models his company might license. Both claim to lead their class. After an hour he has learned almost nothing that would survive contact with his own workload.
The cards are not dishonest. They are written to be skimmed, and they contain real engineering: context limits, evaluation tables, safety notes, licence terms. The trouble is that the numbers are not commensurable — each card wins the comparison its authors chose.
Key takeaways
- A score means nothing without its harness: C-Eval, CMMLU and SuperCLUE test Chinese-language ability on different data from MMLU and agentic task suites.
- Find the training data cutoff first; a model trained after a benchmark was published may be reciting answers rather than reasoning.
- The tokenizer sets both cost and context length, so identical text becomes a different number of tokens in each model family.
- Serving economics belong in the specification: Qwen3.7-Max lists at US$2.50 and US$7.50 per million tokens; DeepSeek V3.1 at ≈US$0.28 and US$1.10 (CGTN).
- Licence terms differ — Apache 2.0, MIT and modified licences are not interchangeable — and they often decide whether a product can ship.
What the headline is measuring
A benchmark is a test, and a test is a choice. Someone chose the questions, the language, the number of examples shown to the model, the scoring rule and whether the model was allowed to use tools. Change any of those and the ranking moves.
The ecosystems have drifted apart. Cards from Chinese labs tend to report C-Eval, CMMLU or SuperCLUE; cards from American and European labs tend to report MMLU and a changing set of agentic suites. These are not translations of one another. C-Eval and CMMLU were built to probe Chinese-language knowledge and reasoning; MMLU was built in English.
A model can be strong on one and ordinary on the other without either card lying.
Self-reported numbers are the norm, and a vendor has every incentive to choose the prompt format that flatters its model. Third-party leaderboards reduce that incentive but introduce their own: they favour submitted models and age quickly.
The practical rule is blunt. A cross-ecosystem comparison of headline scores is mostly invalid. A within-suite comparison, on the same version, run by the same evaluator, is evidence.
Five things to read, in order
- Harness and version. Which suite, which split, how many examples in the prompt, who ran it, and when. A version number matters: suites get revised, and a score from last year's split is not a score on this year's.
- Data cutoff. The date the training data ends. Any benchmark published before that date is a memory test as much as a reasoning test.
- Tokenizer and context. How text is split into tokens, and how much of it the model can hold at once. This drives both the bill and the failure mode on long documents.
- Licence. What you may do with the weights, whether the grant covers commercial use, and whether it survives fine-tuning and redistribution.
- Serving cost. Price per million tokens, but also the output-to-input ratio your workload produces and the latency at your context length.
The first two decide whether the headline number means anything. The last three decide whether you can afford to use the model even if it does.
A card that reports a benchmark and declines to state a cutoff is telling you something by omission. A cutoff later than a benchmark's publication date is not proof of cheating — questions leak into web corpora — but it should lower your confidence in that row.
The same paragraph of Chinese, English or code becomes a different number of tokens depending on the vocabulary, so per-token prices are not comparable across families until converted to cost per document or completed task.
A comparison that misleads
Picture the director's table. Card A leads on a Chinese-language examination suite. Card B leads on an English-language suite that was translated for the Chinese card's audience. The director concludes that A is better for a Chinese customer-support product.
Every step is fragile. The translation may have made the suite easier or harder. The examples shown may differ.
The Chinese suite may test school knowledge far from the code-switching language of real tickets. And neither card reports the actual task: retrieving a policy, refusing an out-of-scope request, handing off to a human.
The honest conclusion is not that one model is better. It is that the two numbers describe different tests, and neither describes the director's problem. The fix is not more benchmarks. It is a small evaluation set drawn from his own tickets, run on both models, scored by his own staff.
The rows the card leaves out
Cards rarely state cost per completed task, latency at load, refusal behaviour on legitimate requests, or reliability when a tool call fails. Nor how performance degrades when the model is quantised to fit cheaper hardware — which is how most buyers will run it.
These omissions are not conspiracies. They are hard to measure and unflattering to publish. But they are exactly the rows a procurement decision needs.
Honest limitations
Model cards are company documents, not audits; the evaluation tables are usually self-reported and the harness details are often incomplete. Benchmark suites themselves are imperfect instruments, and third-party leaderboards carry their own selection effects. The pricing cited here is list pricing from June 2026 and will move. Nothing in a card lets you verify the training data, so claims about contamination or data quality cannot be settled from the document alone.
What readers can do now
- Operators: build a small evaluation set from your own tasks before you read any card, then score the shortlisted models on it.
- Investors: ask for the harness, the cutoff and the licence in the same conversation as the headline score. A team that cannot produce all three has not finished its diligence.
- Researchers: report harness, version, cutoff and licence alongside every result. Without them, a reproduced number is not a reproduced result.