The Harness Tax: Cost per Solved Task Beats the Leaderboard

You can swap the model and hold the harness fixed, or swap the harness and hold the model fixed. Do the first and your cost barely moves. Do the second and it can move by more than an order of magnitude. That asymmetry is the most under-reported fact in AI coding agents in 2026, and it is the one that decides your bill.

Here is what an agent harness actually is, the benchmark numbers that show it — not the model — drives cost, and how to measure the only metric that matters: cost per solved task.

What is an agent harness?

An agent harness is the runtime that wraps a stateless model and turns it into a working agent: the loop that calls the model, the tools it may invoke, how context is assembled and compacted, the permission and safety layer, and the pieces that let it recover from failure. A model is not an agent; the harness is everything that makes it one. As one source-code study of eleven production coding agents put it, "an agent is a model plus a harness."

The harness is where the design work happens

A study that read roughly four million lines across eleven production harnesses (Claude Code, Codex CLI, Gemini CLI, Aider, OpenHands, and others) maps seven canonical subsystems and 29 recurring patterns — and found that no agent runtime imports a general-purpose agentic framework. Every serious system builds its own. Meanwhile the model is the part you now choose from a dropdown.

Why the model stopped being the differentiator

The clearest illustration comes from outside: LangChain reportedly changed only the infrastructure wrapping their model — same weights — and moved from outside the top 30 to rank 5 on TerminalBench 2.0. A separate effort reported a 76.4% pass rate by having a model optimize the infrastructure rather than the model. Berkeley's harness-design lecture states the thesis plainly: the interesting problem "is rarely just what the model should say next. It is what the runtime should let the model see, do, verify, remember, and recover from next."

The numbers: cost spreads 17.5x while pass rate barely moves

If the harness is the variable, it should show up under a controlled test. It does.

FrontierHarness: one model, nine harnesses

Runta's FrontierHarness Eval ran nine harnesses across twelve configurations on 30 identical software-engineering tasks — 360 trials, one model (Kimi K3), one runtime, identical cold-start conditions. The results:

Harness	Pass rate	Median cost / task
Codex	66.7%	$3.47
Claude Code	63.3%	$18.34
DSH Creator	63.3%	$3.28
Pi	60.0%	$2.43
Exo Harness	53.3%	$1.05

Pass rates span 50.0% to 66.7% — a 17-point band. Cost spans $1.05 to $18.34 — a 17.5× spread. Claude Code and DSH Creator achieved the same 63.3% pass rate; one of them paid 5.6× more per pass. And the likely driver is visible in the data: the expensive harness had a 67.8% cache hit rate against a field average weighted at 92.4%.

The overhead benchmark: "OK" costs up to 31,798 tokens

If 17.5× sounds like an outlier, a cleaner measurement makes the same point. A benchmark of eleven coding agents in a single Playwright repo isolated "harness overhead" — the input tokens an agent sends that do not come from your task: its system prompt, tool schemas, instruction files, memory, and environment data. Asking an agent to reply with the single word "OK" burned between 1,894 and 31,798 tokens on the first call, depending on the harness. None of it appears in your terminal, and all of it repeats on every step.

Why the turn loop is where the money burns

Every turn re-sends the whole baseline

A single user query is not a single API call. It is a chain: the model reads the prompt, calls a tool, the result feeds back, and another turn begins. Critically, each turn re-sends the system prompt, re-attaches the tool schemas, and re-reads the entire conversation so far — so token spend scales with your architecture, not with how long your question was, and it compounds across turns. The dominant share of the bill is typically duplicated baseline payload and stale history, not reasoning.

No-action turns and stale history

The excess is not mainly more turns — it is more tokens per turn, including turns where the agent neither edits nor runs anything but still re-reads its full history. A synthesis across harnesses found tokens per solved task ranged roughly 40× (about 28k–37k for one harness versus ~1.15M–1.55M for another) while pass rates stayed inside 38–50%. The model swap moved tokens per solved task only ~1.0–1.3×; the orchestration layer moved it ~40×.

Prompt caching is a cost control, not a nice-to-have

The strongest lever inside that loop is prefix reuse. One vendor run found that swapping only the orchestration layer cut cost per task 41% and tokens per task 38%, lifting completed tasks per million tokens from 54.9 to 92.0. Stable content belongs first in the prompt, the growing conversation after it, and your cache hit rate belongs on a dashboard — one team reportedly runs alerts on it and declares incidents when it drops.

SWE-bench has saturated — so optimize the other axis

On SWE-bench Verified the frontier has run out of room: Claude Opus 5 around 96% (one source 97%), the best independently measured result near 95%, and only a handful of the roughly 79 evaluated models clear 95%. Worse for the scoreboard-watcher: aggregators mix provider-reported scores with calibrated estimates and caution the two are not comparable — GPT-6 appears mainly as estimated entries, not measured runs.

When a benchmark saturates, the signal moves. It moves to the 1–4 hour task bucket, where harness design actually decides success, and it moves to cost. Which brings us to the metric to instrument.

How to measure cost per solved task

Three habits, in order of payoff.

• Instrument the loop like a distributed system. Log tokens at every node, splitting task tokens from baseline tokens. The duplicated system prompt and stale history you find will be a larger number than you expect — the "OK" test above is how you size your floor before you write a single task.
• Measure cost per solved task, not per call. Pass rate and cost are near-independent variables; a metric that ignores one of them lets you buy a 3-point accuracy gain for 5.6× the money. Compare median cost per successful task, at a pinned harness version.
• Watch cache hit rate as a first-class number. A harness at 68% is not "a bit worse" than one at 92%; it is a different cost class, and prompt ordering — not a cheaper model — is how you move it.

Be skeptical of cross-benchmark spreads

The headline multipliers you will read — 5–8×, 17.5×, ~40× — come from different designs and units (per-run cost on five problems, median cost per task, tokens per solved task) and are not directly comparable. One prominent 17.5× figure was audited on the grounds that the dollar amount was reported task spend divided by successful tasks rather than the median cost of successful tasks, and that the run was 360 cells rather than 360 independent tasks. None of that overturns the direction of the finding — a follow-up harness-portability test found no consistent accuracy or cost advantage for a model's native harness — but it does mean "cost per task" needs a definition attached before you trust the number.

Why this is bad news for leaderboard-driven buying

If you choose a coding agent by its model's SWE-bench score, you are optimizing the variable that moves your bill the least. The harness you cannot see — the tools, the compaction, the cache discipline — is where 40× of tokens-per-solved-task lives. That is an uncomfortable conclusion for a market that ships a new leaderboard every week, and it is the right one: the scoreboard ranked the constant, and you are paying for the variable.

FAQ

What is an agent harness?

The runtime around the model: the loop, the tools, context assembly and compaction, permissions and safety, and recovery. "An agent is a model plus a harness" — and the harness is where the engineering and most of the cost live.

Why does the same model cost 17.5x more in one agent than another?

Because the harness re-sends its system prompt, tool schemas, and growing history on every turn, and different harnesses do that at very different cache hit rates. FrontierHarness measured $1.05 to $18.34 median cost per task on one model, with the expensive harness running a 67.8% cache hit rate against a 92.4% field average.

Is cost per token a useful metric?

Not on its own. Compare cost per solved task at a pinned harness version — pass rate and cost vary independently, so a per-token or per-call figure can hide a 5x-overpayment for a few points of accuracy.

Does a model perform best in its own vendor's harness?

Not consistently. A cross-harness portability test found a competing harness matched or beat native harnesses at 0.80×–1.12× cost per pass, with no directional advantage for the native option on either accuracy or cost.

How do I reduce agent cost without a worse model?

Instrument first, then move spend you can see: cut harness overhead at the same pass rate with a leaner loop, order prompts for prefix-cache reuse and alert on hit rate, and budget the ~6k-token system-prompt baseline that repeats per agent per turn in a multi-agent fan-out. Model choice is the last lever, not the first.

The through-line

In 2026 the model became a commodity and the harness became the product — and your invoice is the proof. Pick your agents by cost per solved task, watch the cache, and read a benchmark only after it tells you which harness it pinned. The leaderboard was answering a question you stopped asking. October's releases — worktrees, sandboxing, context compaction — were all harness work, for exactly this reason; the same lesson lands in retrieval, where the plumbing decides the outcome. And if you want a harness that reports its own cost per task instead of hiding it in a monthly bill, that is what we build.