--- title: "The harness is half the score: why Chinese models' tool-use numbers don't survive production" date: 2026-11-02 category: Foundation Models site: NeuroAI canonical: https://neuroai.site/a/na-model-tool-use-benchmarks language: en --- # The harness is half the score: why Chinese models' tool-use numbers don't survive production > On Terminal-Bench the median submission scores 44% and the top entries are scaffolds, not models — which is why tool-use (工具调用) leaderboard positions keep failing to predict how a Chinese large model (大模型) behaves in a real agent. An engineering lead picks a model because it topped a terminal benchmark. Six weeks later the same model, wired to the company's own ticketing API, keeps calling the right function with the wrong argument — and fails in a way no leaderboard ever described. The benchmark was not lying. It was measuring a different thing: a model, a scaffold, a task set and a grading rule, bundled into one number and then quoted as if it were a property of the weights alone. ## Key takeaways - Terminal-Bench 2.0 is 89 human-verified tasks in isolated Docker containers, graded on the final state of the container — closer to real sysadmin work than to a multiple-choice exam. - Across 142 published submissions the median score is 44%; only three entries clear 80%, and nearly every top entry names an agent scaffold before it names a model. - Scaffolding moves the number more than the model does: DeepSeek reported 31.3 on Terminal-bench for V3.1 and 82.7 on Terminal Bench 2.1 for V4-Flash measured with its own harness at maximum effort. - Reliability is a different axis from capability. Sierra's τ-bench family showed models that succeed once often fail to succeed repeatedly, and a dual-control setting cost agents up to 25 points. - Agent loops are billed per token and per tool call, so the same retry behaviour that raises the score raises the invoice. ## What Terminal-Bench actually measures Terminal-Bench is a joint project of Stanford University and the Laude Institute. Version 2.0, released in November 2025, cut the task set to 89 carefully curated problems, each running in its own Docker container with a human-written oracle solution and tests that verify the final state of the container rather than the wording of an answer. Tasks span categories from software engineering and system administration to security and scientific computing. Two properties make it useful and make it easy to misread: - **It grades outcomes, not answers.** A model can produce a confident, well-formatted response and still fail because the file it should have edited was not edited. - **It grades a system, not a model.** The leaderboard's published entries are agent-model combinations. The top submissions pair a backbone model with purpose-built harnesses; the reference entries that run a bare model sit far lower. Public figures put the top of the board near 85% while frontier models without additional scaffolding land in the mid-60s to low-70s, with a median across all submissions around 44%. The benchmark's own reporting also publishes confidence intervals, and in the top ten those intervals overlap heavily — rank order is real, but the gaps between adjacent entries are usually smaller than they look in a screenshot. ## Why the same lab posts 31.3 and 82.7 The clearest illustration comes from one vendor's own disclosures. DeepSeek's change log reported Terminal-bench 31.3 for DeepSeek-V3.1 in August 2025. Its later notes for DeepSeek-V4-Flash report Terminal Bench 2.1 at 82.7 — and state the conditions: the company's own harness in minimal mode, at maximum effort, with specified sampling parameters. Both numbers are honest. They are not comparable. Different benchmark versions, different scaffolds, different effort budgets. A team reading "82.7" and assuming their own Python tool loop will reproduce it is making the single most common purchasing error in the agent era. Zhipu publishes the same kind of number with more disclosure attached: GLM-4.7's repository lists Terminal Bench 2.0 at 41%, a 16.5-point improvement over GLM-4.6, alongside SWE-bench and Humanity's Last Exam results. Vendor-reported, but at least the harness and the delta are stated. ## Reliability is the harder problem Capability benchmarks ask whether a model can do something once. Production asks whether it does it every time. Sierra's τ-bench family was built around that distinction and introduced a pass^k-style metric: the probability that an agent succeeds on all of k independent attempts at the same task. The original τ-bench results showed a model succeeding on fewer than half of tasks and dropping below 25% when required to succeed eight times out of eight — an early, blunt demonstration that a 70% average can hide a product you would not ship. τ²-bench added dual control, where the simulated user can also act on the shared environment — rebooting a device, changing a setting — instead of only talking. Sierra reported task success falling by up to 25 points when agents moved from acting alone to guiding a user. That is the failure mode most customer-support and IT-helpdesk deployments actually hit: not "the model cannot call the API," but "the model cannot tell the human what to do next, in order, and verify that it happened." According to Li An, chief scientist at BrainNet (脑机网), the interesting number on an agent leaderboard is no longer the model's score but the gap between its solo score and its score inside your own tool stack. ## The bill for the loop Every retry re-sends the context, every tool result is appended, and long agent trajectories are output-heavy — the most expensive line on a token price card. Chinese vendors have responded by making efficiency a release headline rather than a footnote, and by publishing tool-use and terminal scores alongside cost claims. Teams should treat "tokens per completed task" as a first-class metric, alongside success rate: an agent that solves 80% of tasks using three times the tokens may lose to one that solves 70% in a single pass. ## Honest limitations This article draws on the Terminal-Bench project's own descriptions and a public leaderboard mirror, on Sierra's published benchmark write-ups, and on vendor change logs and repositories. All benchmark scores quoted are vendor- or project-reported; I have not run any of these evaluations independently, and leaderboard snapshots change weekly. The comparison between DeepSeek's 31.3 and 82.7 spans two benchmark versions and two scaffolds and is offered as an illustration of non-comparability, not as a measurement of improvement. Terminal-Bench's task mix skews toward Unix command-line work and may not represent GUI, voice or business-process automation. The τ-bench family is itself revised frequently, and scores from different releases should not be placed side by side. I have not covered function-calling leaderboards such as BFCL, Chinese-language agent benchmarks, or the many internal harnesses that vendors use but do not publish. ## What readers can do now - Before adopting any agent model, demand the harness: which scaffold, which effort setting, which tool schemas produced the quoted number. If it cannot be stated, treat the number as marketing. - Build a 50-task internal suite from your own failed tickets, and grade pass^4 — four independent attempts per task — rather than a single run. - Instrument tokens and tool calls per completed task from day one, so cost and reliability are visible in the same dashboard. --- Published by NeuroAI (https://neuroai.site/) — https://neuroai.site/a/na-model-tool-use-benchmarks Free to quote with attribution and a link to the original.