How many times in a row does an AI agent get the same task right?

Fewer than its benchmark score suggests, and the gap is already published. On Sierra's tau-knowledge banking domain the leading model in May passed 37.4% of tasks on one attempt and 20.6% on all four. The leaderboard we read this week carries the first number and not the second.

Those two figures come from the same runs. One asks how the average attempt went. The other asks whether every attempt at a task went that way.

What is pass^k, and what does it measure that a score does not?

It measures consistency. tau-bench, the Sierra benchmark that introduced the metric in 2024, defines pass^k as "the chance that all k i.i.d. task trials are successful, averaged across tasks", estimated from n trials per task as the mean of C(c,k)/C(n,k) for c successes. Its sibling pass@k asks whether at least one of k trials worked, which is the discovery question.

The ordering is the whole mechanism. A one-run score averages across every trial at once, so a task that failed three times out of four and a task that failed once out of four both disappear into the same mean. pass^k multiplies inside the task first and averages afterwards, so a task that cannot be done four times running is counted as not done.

The spread that matters is between tasks, not between runs.

How far does the score fall when the same task runs four times?

By a third to two thirds, on every published pair we could find. The first two rows below come from the tau-bench leaderboard, which still reports pass^1 through pass^4 for 2024 models. The second two come from Sierra's tau-knowledge post of 13 May 2026, on a banking domain of 698 policy documents across 21 product categories, where a task needs 18.6 documents and 9.5 tool calls on average.

Benchmark and model	pass^1	pass^4	pass^4 if runs were independent
tau-bench retail, Claude 3.5 Sonnet (Oct 2024)	69.2%	46.2%	22.9%
tau-bench airline, Claude 3.5 Sonnet	46.0%	22.5%	4.5%
tau-knowledge banking, GPT-5.2 high (launch)	25.5%	9.3%	0.4%
tau-knowledge banking, GPT-5.5 xhigh (May 2026)	37.4%	20.6%	2.0%

The fourth column is ours, and it is arithmetic rather than a measurement: pass^1 raised to the fourth power, which is what pass^4 would be if each run were an independent draw at the published rate. Checked by Unity Horizon on 6 November 2026.

Why does the drop stop where it does?

Because the failures are not per-run noise. Four runs of a 37.4% agent would all succeed 2.0% of the time if every run were a fresh coin, and the published figure is 20.6%, ten times that. On retail the published pass^4 is twice the coin figure. On banking at launch it is twenty-two times.

The failures sit on particular tasks instead. The agent does some of them every single time and misses others on nearly every attempt, and the mean of those two populations describes neither of them. Fit the crudest two-point model to the May banking pair and roughly a fifth of tasks pass on all four runs while the rest pass about one run in five. That fit is an illustration and not a measurement. The direction is forced by the two published numbers.

That gives pass^4 a plain reading. It is close to the share of tasks the agent owns outright, and the distance back up to pass^1 is the flaky band. The band is 16.8 points on banking in May and 23.0 points on tau-bench retail.

A score is an average over tasks. A workflow is one task, over and over.

This is a different problem from whether a two-point gap between two models on a benchmark is real, which is about noise in the estimate. The estimate here is fine, and it is estimating the wrong thing.

Does running it again fix it?

On the flaky band, often. On the rest, no, which is why both metrics exist: pass@k rises with k because it needs one success, and pass^k falls with k because it needs every one. Price a retry policy against the first and a staffing plan against the second.

Retries carry consequences too. Only the model call is safe to repeat, as the three layers of an agent that retry on their own showed, and nothing between the model and the tool deduplicates a write. So the move that pays is routing rather than retrying, and the measurement hands you the routing for free: the repeated sampling that produces a pass^k figure is the same repeated sampling that gives an agent a usable uncertainty signal, where disagreement across ten samples scores 0.790 AUROC against 0.698 for asking the model whether it is right.

What does this mean for a workflow that runs 240 times a month?

It decides who you hire. We built OFFSET, which files declarations directly to the ΑΑΔΕ portal, and the worked estimate we use for that kind of filing is 30 properties averaging 8 stays a month, roughly 240 declarations. Estimated from the fields on the form, not timed with a stopwatch.

Suppose your measured pass^1 on that workflow is 95%. You expect about 12 failures a month either way. What pass^1 cannot tell you is which of two months you are about to have. Twelve different properties failing once each is a retry queue and a quiet Monday. One property whose address field breaks the form on every attempt, plus a handful of scattered misses, is a property that comes off the agent until somebody fixes the field mapping and a person files those eight stays by hand.

Same headline number. Two different staffing decisions, and the figure that separates them is pass^4 per task shape, which nobody publishes for your tasks.

Why is the reliability number missing from the leaderboards?

Because it costs k times as much and it is a separate command. The released tau2-bench benchmarks default to one trial, where task success is pass@1, and a second command runs three trials to report pass^3. One trial of the text benchmark is 900 calls, so pass^3 is 2,700.

The tau3-Banking leaderboard we fetched on 6 November 2026 has a Pass^2, a Pass^3 and a Pass^4 column, is sorted by Pass^4, and carries no value in any of its 27 rows. GPT-5.5 sits in it at 44.6% pass^1 with nothing beside it. That 44.6% is not the May figure improved, either: a July 2026 release regraded the banking domain and says outright that "scores computed before and after this release are not comparable on banking_knowledge". The column is there. The budget for it is not.

METR's time-horizon work turns the same knob from the other end. Raise the required success rate from 50% to 80% and the task length a model can handle falls roughly five times, 59 minutes to about 15 for Claude 3.7 Sonnet. METR's note of 20 March 2026 says the 80% figures are the fragile ones, and that alternative curve fits move them by up to 100%. The multiple is arguable, the direction much less so.

Who gains from a one-run score, and who pays?

Anyone publishing a model number gains, because one trial is the default and the cheapest honest run available. Teams with four or five task shapes gain too, because sixteen or twenty runs is an afternoon and they end up knowing something their vendor does not.

The bill lands on the operator who meets the same failing case every month while the dashboard reports the average, and on the buyer who approved a pilot in which each case was tried once. A pilot that runs every case once measures capability. A month of production measures reliability, and by then the contract is signed.

What would we measure first?

Four runs of every task shape, with the inputs frozen. The method a reader can repeat by Friday:

• list the shapes by the work being done, a cancellation or a filing for a property with no VAT number, rather than by which customer asked
• pin the input: same record, same data, same starting state
• run each shape four times and record four of four, some of four, or none of four
• hand the none-of-four shapes to a person and leave them there until the shape itself changes, rather than until the next model ships
• re-run the some-of-four shapes after any prompt or model change, because that band is where a change shows up first

Three buckets per shape fits on one page. Put that page in front of whoever signs for the automation.

What we have not published is a pass^4 figure of our own, on any agent we built, so read the arithmetic above as arithmetic over other people's measurements. The 240 declarations is an estimate from the form, and the two-point fit is a fit. We also read that leaderboard the way a crawler reads it, so if those columns fill in somebody's browser, we have not seen them.

Which of your agent's task shapes has been run four times on the same input, and who has all four results? If the shapes are not written down yet, that is where we would start.