A new large model (大模型) lands with a table: 92% here, first place there. The numbers decide headlines and budgets. Most of them measure one narrow thing, and almost none of them measure your job.
Here is how to read the scoreboard without being sold.
The knowledge test: MMLU
MMLU (Massive Multitask Language Understanding), from Hendrycks et al. (2020), is a multiple-choice test across 57 subjects — elementary math through law — about 15,900 questions. In 2020 models scored near the 25% guessing floor; frontier models now sit near 90%. That is the problem: MMLU has largely saturated, so a 2-point gap between two top models means little.
The grade-school math: GSM8K
GSM8K (OpenAI, 2021) holds 8,500 elementary math word problems needing 2–8 arithmetic steps. Frontier models clear 95%, so it too is saturated and no longer separates the best systems.
The real coding test: SWE-bench
HumanEval (OpenAI, 2021) checks whether 164 small Python functions pass unit tests — useful, but easy for modern models. SWE-bench (Princeton, 2023) is harder: 2,294 real GitHub issues across 12 Python repositories, where the model must read a codebase and produce a patch that passes the project's own test suite. SWE-bench Verified is a human-checked ~500-issue subset released by OpenAI and Princeton in 2024. It became the dominant coding benchmark — until contamination concerns surfaced. According to 2026 reporting, OpenAI signaled it would move to deprecate SWE-bench Verified after flagging that several frontier models showed signs of training-data contamination on it. The lesson: a benchmark's life ends when models start memorizing it.
The PhD test: GPQA
GPQA (Rein et al., 2023) is "Graduate-Level Google-Proof Q&A": 448 multiple-choice questions in biology, physics, and chemistry written by PhDs so that non-experts with web access score only about 34%, while domain PhDs score roughly 65%. The 198-question "Diamond" subset is the most-cited. It is one of the few tests still spreading frontier models apart rather than clustering them near the ceiling.
The math ladders: AIME and MATH
Beyond GPQA, two math ladders matter. AIME (American Invitational Mathematics Examination) is a real high-school contest that vendors quote because it is hard and public. MATH (Hendrycks et al., 2021) holds 12,500 competition problems across seven subjects with strict exact-match grading. Both saturate less than GSM8K but are still gameable, so always read the variant and the date.
The human test: preference leaderboards
Chatbot Arena (now LMArena), from UC Berkeley, ranks models by blind pairwise votes — closer to "which answer do people prefer" than "which passes a test." It catches usability that static tests miss, but the votes shift with who is voting and are easy to nudge with release-day buzz. Treat it as a sentiment gauge, not a spec.
When the old tests die: Humanity's Last Exam
Humanity's Last Exam (2025) exists for exactly this reason: its authors noted that models now exceed 90% on popular tests like MMLU, which limits what those tests can still tell us. Its 2,500 expert-written questions, many with images, are designed to resist quick web lookup. Watch for it becoming the next "does this model actually think?" bar.
A practical rule of thumb worth repeating: if a vendor leads only on a benchmark you have never heard of, that is usually the point — they chose the test where they win. A confident claim names the benchmark, the exact variant, the date, and whether a contamination check was run. Anything less is a slogan dressed as a measurement.
Three traps in any score table
- Different metrics, same name. "SWE-bench" and "SWE-bench Verified" are different tests; a model's 80% on one is not comparable to 80% on the other.
- Saturation hides the truth. Once a test nears 90–95%, small differences are noise. Watch for which benchmark a vendor leads on.
- Contamination and private sets. Public tests leak into training data. Trust benchmarks with held-out or private evaluation sets, and discount any number without a contamination check.
What readers can do now
- Name the exact metric. Ask "which variant, which harness?" before believing a claim.
- Run your own mini-eval. Build 20–50 tasks from your real work; leaderboard rank is a weak predictor of your use case.
- Watch for saturation and contamination. Prefer benchmarks the model has not memorized, and re-test quarterly.
Honest limitations
This guide summarizes benchmark definitions from published papers and secondary explainers; I have not re-run any evaluation. Scores shift as harnesses and contamination checks change, and "best" depends entirely on your task. I treat vendor-reported numbers as claims to verify, not facts.
Key takeaways
- MMLU and GSM8K are largely saturated; GPQA Diamond and SWE-bench Verified still separate frontier models.
- Always name the exact benchmark variant and metric — "SWE-bench" ≠ "SWE-bench Verified."
- Benchmarks decay through saturation and contamination; your own small eval beats any leaderboard for real work.