Intelai
The Benchmark Number Is a Claim About the Harness
Claude 3.7's TAU-bench score came with a footnote about scaffolding. That footnote is the most useful thing in the announcement.
In February 2025, Anthropic announced Claude 3.7 Sonnet (opens in a new tab) and, in the same post, Claude Code. Buried in the benchmark appendix was a note about how the TAU-bench score was produced.
The note said the score came with a prompt addendum instructing the model to use a planning tool, and that the maximum number of steps was raised from 30 to 100 to accommodate the extra thinking. The company also re-ran the previous model on an updated dataset so the comparison would be fair.
That footnote is the most useful thing in the announcement, and it is the reason I stopped reading benchmark tables the way the vendors print them.
What TAU-bench actually measures
TAU-bench came out of Sierra in June 2024, and the paper (opens in a new tab) is worth reading if you deploy agents against real systems. It sets up a simulated user and a set of tools in realistic domains, then asks the agent to complete a task over multiple turns. The user has a goal statement and a set of instructions the agent cannot see. The tools have rules. The agent has to ask, act, recover, and finish.
The metric that matters is not a single success rate. TAU-bench introduced pass^k, which measures whether the agent succeeds on the same task across repeated trials. That is a reliability measure, and it is much closer to what a program office actually cares about than a top-line average.
An agent that completes a task 70 percent of the time sounds fine until you notice it completes the same task 70 percent of the time on every attempt, which means one in three runs of your workflow needs a human to catch it.
The footnote is the finding
If a benchmark score changes when you change the prompt and the step budget, then the score is not a property of the model. It is a property of the model plus the harness, and the harness is a set of choices somebody made.
That is not cheating. It is the honest condition of agent evaluation, and the fact that Anthropic disclosed it puts them ahead of most vendors who publish a number and a logo.
But it does mean you cannot compare two numbers from two vendors unless the harnesses match, and they almost never do. One allows 30 steps. One allows 100. One gives the agent a planning tool. One gives it a scratchpad. One grades with a model, one grades with a script.
A benchmark score is a claim about someone else's harness, tested on someone else's tasks.
The useful question is not which model scored higher. It is which of those harness choices resembles the one you are about to deploy, and what happens to the score when you remove the ones you cannot have.
What this means for a federal deployment
Your environment will not look like the benchmark. Your tools are older, your user is a contracting officer with a deadline, and your failure cases involve documents that were scanned badly in 2011.
Three things follow.
First, a vendor's completion rate on airline booking tells you almost nothing about procurement workflows. The domain is different, the tools are different, and the cost of a wrong action is different.
Second, the scaffold you build matters as much as the model you pick. The same model with a planning step, a retrieval layer, and a refusal path will beat a better model with none of those. Anthropic's own footnote is evidence for that, from the vendor with the most to gain from the opposite conclusion.
Third, reliability has to be measured per task, not in aggregate. A 90 percent average across ten task types can hide a task type that fails every time, and that task type is usually the one your program depends on.
What to measure instead
Build a small evaluation set from your own work. Twenty tasks is enough to start. Pick the ones that a new hire would get wrong in the first month, because those are where the agent will fail in a way nobody notices.
Run each task at least three times. One run measures capability. Three runs measure whether you can rely on it.
Score four things, in this order. Did it complete the task. Did it complete it without an action you would have to explain to a security reviewer. Did it escalate when it should have. What did it cost, including the retries.
Keep a failure taxonomy. Not "it was wrong," but which step failed and what the agent believed at the time. The taxonomy is what you hand the vendor, and it is also what you hand your own engineers when the model gets swapped and the behavior changes.
The part vendors will not volunteer
Every model release will come with a chart that shows improvement, and the chart will be true in the sense that the number was produced by a real harness on real tasks. It will also be irrelevant to your deployment in a way that no one is lying about.
The organizations that get good outcomes from agents are not the ones that pick the highest number. They are the ones that built a test set before they bought anything, ran it against three vendors, and kept running it after the contract was signed.
That is unglamorous work, and it is the only work that tells you whether the thing you deployed still works after the next model release quietly changes its behavior.
Buy the harness you can test. The model is the part that changes.