An agent that passes once hasn’t passed

A demo is one sample

When an agent finishes a task in a demo, what you've learned is that the task can be done. You haven't learned how often it gets done. For anything you'd put in front of users, how often is the question that matters.

The numbers on this gap are a few years old but they still hold up. In the Codex paper, OpenAI's model solved 28.8% of the HumanEval coding problems with one attempt each. Give it 100 attempts per problem and count a problem as solved if any of them passed, and the figure goes to 70.2%. Same model, same problems. The bigger number answers "can it ever do this" and the smaller one is closer to "will it do this when I ask."

Agents widen that gap because there are more places for a run to go differently. Tool responses change, steps happen in a different order and a small difference early on can send the rest of the run somewhere else.

Run it again, on purpose

τ-bench is the clearest example I know of measuring this directly. It gives agents customer service tasks in simulated retail and airline settings, runs each task several times and reports pass^k: the chance that all k attempts at a task succeed, not just one of them. GPT-4o got 61.2% of retail tasks right on a single attempt. Its pass^8 in retail was under 25%. A lot of the tasks it could solve, it couldn't solve every time.

Setting the temperature to zero doesn't get you out of this. One study ran five LLMs configured to be deterministic ten times each across eight tasks, and accuracy moved by up to 15% between runs. Thinking Machines Lab sent the same prompt 1,000 times at temperature 0 and got 80 different completions, and traced most of that to how inference servers batch requests under load. They showed it can be fixed. Unless your stack does that, assume reruns differ.

So when I design trials I vary the things a real user would vary (wording, starting state, slow or failing tools) and keep the grading fixed, then run every case more than once. I also write down how bad each failure was. An agent that fails by asking a clarifying question and one that fails by deleting the wrong file both count as a fail in a pass rate. They shouldn't be weighed the same.

Twenty similar cases aren't twenty cases

The other trap is counting evidence you don't have. If ten of your test cases are small rewordings of the same task, they'll tend to pass or fail together. That's closer to one data point than ten.

Anthropic put numbers on this in their writeup on statistics for evals, which summarizes a paper by Evan Miller. When questions come in related groups, like several questions about one passage, the honest error bars on some popular benchmarks came out more than three times as wide as the naive ones. A score can look precise and still be much less certain than it seems.

Software testing has lived with a version of this for a long time. Google reported in 2016 that almost 16% of its tests showed some flakiness, passing and failing on the same code, and that about 84% of the pass-to-fail transitions they saw involved a flaky test. One green run was never as strong a signal as it looked. Agents just turn the noise up.

One habit from regression testing carries straight over: keep the cases you've already debugged separate from the ones that measure new capability. Once you've tuned against a failure, passing it tells you the fix held. It doesn't tell you much about the next problem.

What a reliability claim should say

When I say an agent works, I want the claim to come with the boring details. Which tasks, which tools and environment, how many runs per task and how many genuinely different scenarios those runs covered. Then the pass rate with a range around it, and the worst failures named individually instead of folded into a percentage.

That's more work than recording a demo. But a demo shows the agent on a good run, and users don't only get the good runs.