Terminal-Bench 4.0 and honest comparisons of coding agents
Terminal-Bench 4.0 is a new task set, not a harder 2.1, and vendor and independent scores already differ. How to read the numbers before they reach a decision.
Terminal-Bench has become one of the numbers that shows up in model launches and in slides comparing coding agents. Version 4.0 is now the one being quoted, and it is a different test from the version most teams remember. Scores on it are much lower than on 2.1, the same model already has several different 4.0 scores depending on who ran it, and some launch posts list results from more than one version side by side. None of that is a problem with the benchmark. It is a reason to read the numbers carefully before they influence a tooling or model decision.
The context: what Terminal-Bench 4.0 is
The Vals AI Terminal-Bench 4.0 page, updated on 13 October 2026, sets out how 4.0 differs from 2.1:
- New tasks. All 66 tasks are new; none overlap with the 89 tasks in 2.1.
- Broader scope. Roughly three quarters of 4.0 lies outside traditional software engineering, in science, ML systems, operations, hardware, security and media. Tasks are tagged by domain and by an expert time estimate rather than easy, medium or hard.
- Long time limits. Every task allows the agent 8 hours.
- Strict scoring. A model must pass a task's full test suite to get credit; there is no partial credit. Vals reports avg@3: three full runs, each scored pass@1, averaged.
- A different harness. Vals ran 2.1 with the Terminus 2 harness and 4.0 with mini-swe-agent, a minimal agent whose only tool is bash.
Vals states the consequence plainly: "Because the task set is entirely different, scores are not comparable across versions."
Artificial Analysis made the same move. Version 1.5 of its Coding Agent Index, from September 2026, replaced Terminal-Bench 2.1 with 4.0 and averages three components with equal weight: DeepSWE v1.1 (113 tasks), Terminal-Bench 4.0 (66 tasks) and SWE-Atlas-QnA (124 tasks), each attempted three times. It also checks Terminal-Bench 4.0 attempts for reward hacking, such as editing tests or fetching a reference solution, and scores those attempts zero, because 4.0's tasks, tests and solutions are public and attempts run with internet access. How much leakage can inflate coding scores is shown in SWE-Bench Pro Verified: when coding agents were reading the answers.
Analysis: what to make of it
Version mixing is easy to do by accident
The DeepSeek-V4.1-Flash model card lists Terminal-Bench 2.1, 3.0 and 4.0 in one table, at 90.6, 30.0 and 31.2 respectively. The card labels each version clearly, so this is not misleading in itself. But a summary that quotes "around 90 on Terminal-Bench" next to another model's 4.0 score would be comparing different tests. Any comparison should name the version, and only compare like with like.
Vendor and independent numbers already disagree
Take three models that appear in both vendor material and on the Vals leaderboard, all on 4.0:
| Model | Vendor-reported | Vals AI (mini-swe-agent) |
|---|---|---|
| Claude Sonnet 5.5 | 70.6% (Anthropic) | 64.14% |
| Claude Haiku 5.5 | 39.2% (Anthropic) | 36.36% |
| DeepSeek V4.1 Flash | 31.2 (DeepSeek, own harness) | 19.70% |
The Haiku 5.5 announcement of 7 October reports the Anthropic figures. DeepSeek's card says its code agent benchmarks, Terminal-Bench included, use the Minimal mode of its own DeepSeek Harness at maximum reasoning effort. Vals runs every model through the same minimal harness. Neither set of numbers is wrong; they measure different combinations of model, harness, settings and runs. The gap is largest for the model whose vendor used its own harness, which fits the wider point that the harness is part of what is being measured, as covered in the harness sets the bill.
Provider-side details can move a score
Vals notes something most leaderboards do not show. For five Anthropic rows, some task attempts were served by an older model through provider-side fallback. Opus 5.5, top of the board at 65.15%, had 22 of 198 attempts served by Opus 5 or Claude Opus 4.8; counting those as failures lowers it to 58.08%, behind Sonnet 5.5 and GPT-6 Astra. Sonnet 5.5 had 4 such attempts and drops from 64.14% to 62.63% on the same basis. That ordering change comes from how the service behaved during the run, not from the models' ability on the tasks.
A low score is not a bad model
Because most of 4.0 is outside traditional software engineering, a model that scores modestly may still be well suited to everyday repository work. Terminal-Bench 4.0 measures long, demanding terminal tasks across many domains. It is a useful signal for autonomous agent work, and a weaker one for code completion or review.
What it means for engineering and security leaders
Benchmarks are a starting point for a shortlist, not a decision. When a number reaches a planning document, it should carry four labels: the benchmark version, the harness, who ran it, and the date. Without them it cannot be compared with anything else.
For decisions that matter, such as choosing a default model for a team or approving an autonomous agent for a class of work, run a small evaluation on your own repositories and tasks, with the harness your developers actually use. The cost of that is small next to the cost of a wrong default. For small open-weight models specifically, see open-weight models as coding sub-agents.
Practical takeaways for this week
- Check any model comparison in circulation for mixed benchmark versions, and correct or label it.
- Record version, harness, source and date next to every benchmark score you cite internally.
- Prefer independent runs with a common harness for comparison, and treat vendor figures as claims to check rather than as a baseline.
- Read leaderboard notes for fallback, refusals and reward-hacking rules before ranking close results.
- Keep a small internal task set from your own repositories and rerun it when you consider a model or harness change.