---
title: "Why the same AI model gets two different scores on the same benchmark"
date: 2026-11-16
category: Foundation Models
site: NeuroAI
canonical: https://neuroai.site/a/na-model-terminal-bench-harness-gap
language: en
---

# Why the same AI model gets two different scores on the same benchmark

> One model, one benchmark version, two numbers. The gap between vendor-reported and independently-run agent scores is now bigger than the gap between rival models.

Pick a model. Pick a benchmark version. Then ask two organisations to run it, and watch the number move.

This month it happened twice, in public. Tencent reported that Hy4 scores **44.4** on Terminal-Bench 4.0. The independent leaderboard run by Snorkel AI lists the same model, on the same task set, at **39.1**. Moonshot's Kimi K3.1 Pro appears at **65.7** in its own reporting, **63.6** on one aggregator and **60.9** on another.

Nobody is lying. That is the uncomfortable part.

## What Terminal-Bench actually measures

Terminal-Bench scores an agent on real command-line work inside containers: install this, debug that, make the test suite pass. The metric is a resolution rate — the share of tasks finished successfully.

Version 4.0 is deliberately hostile. It trimmed the 3.0 set from 74 tasks to 66, removing eight and revising 20, and added nothing. It is, by design, a rolling target that gets harder as agents catch up to it.

But here is the detail that decides everything: **the score is not a property of the model alone.** It is a property of a model plus a harness (运行框架) — the loop that decides what to show the model, when to run a command, how to recover from an error, and when to declare victory.

The same checkpoint driven by Claude Code, Codex, Grok Build or mini-SWE-agent produces different numbers, because those harnesses feed context differently, retry differently and stop differently. A model that looks mediocre inside a thin scaffold can look excellent inside a well-tuned one. Half of what the leaderboard measures is scaffolding.

## Four reasons two numbers differ

When you see two scores for one model, the explanation is almost always one of these:

- **Different harness.** Snorkel's board lists Opus 5.5 at 64.8% with Claude Code and 59.6% with mini-SWE-agent. Same model. Five points.

- **Different effort setting.** Many entries run at "max" or "xhigh" reasoning effort, and a board will often carry several rows for one model. Opus 5 appears at 53.9% at xhigh effort on one board and 49.0% in a default configuration on another.

- **Different runner.** Vendor self-reports, an independent evaluator like Vals AI, and a platform leaderboard each produce their own numbers. Vals runs everything through Mini-SWE-agent and averages three runs; a vendor may report one.

- **Different version.** Terminal-Bench 2.1 and 4.0 share a name and nothing else. Vals made 4.0 its coding test on September 25, 2026, replacing 2.1, and its own note says the two do not compare.

That last point produces the single most misleading artefact in current model marketing. Meituan's LongCat-2.5 card reports **88.4** on Terminal-Bench 2.1 — within a point of Claude Opus 5's cited 89.1 — and **38.5** on Terminal-Bench 4.0, where the same card lists Opus 5 at 53.9. Both statements are true. Quoted alone, either one misleads.

## The column nobody quotes

The most useful thing on the Snorkel board is not the score. It is the cost column sitting next to it.

Look at the top of that table. Claude Opus 5.5 reaches 64.8% while burning **8.0 billion tokens** and roughly **US$4,700** across the run. Kimi K3.1 Pro reaches 60.9% on **3.2 billion tokens** and about **US$1,100** — nearly the same score for a quarter of the money. Hy4 lands at 39.1% for roughly **US$300**.

Put differently: the top model costs about 15 times as much as the cheap Chinese entry for a score roughly 1.7 times higher. Whether that trade is good depends entirely on what a completed task is worth to you — and no leaderboard can answer that, because it depends on your business, not on the benchmark.

This is where the Chinese open-weight (开源权重) releases of the past fortnight change the calculus. If a model completes 60% of tasks at a quarter of the price, the rational move for most teams is to route the cheap model at everything and escalate only the failures to the expensive one. That is a systems decision, and the leaderboard is not built to express it.

## Why vendors keep quoting the flattering number

There is no conspiracy here, just incentives stacked in one direction.

A vendor controls which harness it tests in, how much reasoning effort it grants, how many runs it averages and which version of the suite it names in the blog post. Every one of those choices is legitimate. Every one of them also tends to move the number up. An independent evaluator has the opposite problem: it must hold the harness constant across dozens of models, which guarantees that no single model gets its ideal setup.

Add a second layer: benchmarks are now released faster than they can be validated. Version 4.0 arrived in August 2026; the comparison sets quoted beside it were assembled before several competing models existed. Reviewers have noted that some models posting higher 4.0 scores are absent from vendor tables altogether — not out of malice, simply because the table was built earlier.

The result is a market where a benchmark (评测基准) name functions as a brand. A score of 88 on Terminal-Bench travels further than 38.5 on the current version of the same suite, and both can describe one model in one week.

## Honest limitations

- The numbers in this piece come from public leaderboards and model cards read on November 15–16, 2026. They move. Several models cited here have newer releases that were not independently scored at the time of writing.

- Comparing across boards is itself unsound: Snorkel, Vals AI and aggregator sites use different harnesses, effort settings and averaging rules. The disagreement is the story, not a defect in any one board.

- Cost figures reflect token pricing at the time of the run, not subscription or committed-use discounts.

- Terminal-Bench measures terminal-shaped software work. It says little about writing quality, customer support, medical triage or any task where "resolved" is a judgement call.

- This article did not run any evaluations. It compared published numbers and their stated conditions.

## What readers can do now

- **Never accept a benchmark number without four qualifiers**: harness, effort setting, runner, version. If a launch post omits any of them, assume the most flattering value was used.

- **Build a ten-task private set from your own backlog.** It will not be statistically rigorous, and it will still outpredict any public leaderboard, because it measures the work you actually pay for.

- **Score cost per completed task, not per token.** Multiply resolution rate by price per run. A model that finishes 40% of tasks for US$300 can beat one that finishes 65% for US$4,700 — depending on whether a failure costs you a retry or a customer.

---

Published by NeuroAI (https://neuroai.site/) — https://neuroai.site/a/na-model-terminal-bench-harness-gap
Free to quote with attribution and a link to the original.
