The harness sets the bill: why the same model costs more in some agents

Most discussions of AI coding costs focus on the model: which one is cheapest per token, which one scores highest. Recent benchmark results point to something teams often overlook. The same model, run inside different agent software, can produce different results at very different costs. The software around the model, usually called the harness, decides how many calls it makes, how much context it sends and when it stops. This article looks at the evidence and what it means for choosing and measuring coding tools.

The context: a benchmark for agents, not models

Artificial Analysis's Coding Agent Index measures complete agents, a harness plus a model, rather than models alone. Version 1.5 of the index combines three task sets with equal weight: DeepSWE v1.1 with 113 tasks, Terminal-Bench 4.0 with 66, and SWE-Atlas-QnA with 124. Each result is pass@1 averaged over three attempts, and the index also reports cost per task.

On 10 October, Reason Machines published results for its Reason Agent 1.1 harness. On the same model, GPT-6 Astra, it reports an index score of 65.1 against 62.5 for Codex, at 15–23% lower cost per attempt across the 303 tasks. More striking is a broader observation in the same post: the same model can cost up to five times more in one harness than in another.

Why harnesses differ so much

A harness makes many decisions that affect cost:

• How much context it sends. Some read whole files; others search first and read only what they need.
• How many steps it takes. Some plan, act and check in many small calls; others do more per call.
• Whether it keeps the prompt prefix stable. A harness that keeps tool lists and instructions fixed gets more cheap cache reads.
• When it stops. Some retry failing tests repeatedly; others stop and report.
• Which models it uses for which steps. Sending simple steps to a smaller model can cut cost without changing the result.

None of these show up in a model's price per token.

Analysis: what to make of it

Vendor benchmarks need caution. The Reason Machines figures are the vendor's own results on its own product. The independent index and its published method make them easier to check, but a single release post is not a settled comparison.

The direction is still credible. A spread of several times in cost for the same model is consistent with how harnesses work. Anyone who has watched two agents attempt the same task has seen one read three files and the other read thirty.

Score per dollar matters more than score. A few points of benchmark score rarely matter as much as a several-fold difference in cost per task, especially across a whole engineering organisation.

Benchmarks are not your codebase. Public tasks reward particular behaviours. Your repositories, tests and conventions may favour a different harness.

What it means for engineering and security leaders

When evaluating AI coding tools, compare the tool and model together, on your own work, and measure cost per completed task alongside quality. Reviewing model price lists alone will mislead you.

It also means cost control is partly a tooling decision. Moving a team to a different harness, or changing its settings, can change spending as much as changing models.

Practical takeaways for this week

• When you compare coding tools, record cost per task, not only the result.
• Run the same small set of real tasks from your own repositories through each candidate, with the same model.
• Track tokens per session, cache reads and number of model calls per task for each tool in use.
• Treat vendor benchmark claims as hypotheses to check, and look for the method behind them.
• Revisit tool settings, such as how much context they load, before switching models to save money.

Where SpanDock fits

Through the SpanDock model proxy, in every plan, each call records the requested model, the model that answered, tokens and cost, and the Models page ranks the models your tools used by spend or cost per million tokens. On Standard and up, SpanDock Mind chooses a cheaper model for a new conversation when it is good enough, and per-person budgets set a limit for a day, a week or a month.