Why does the same prompt give a different answer at temperature 0?

Because the request never reaches the GPU on its own. It is batched with whatever else arrived in the same few milliseconds, batch size changes the order the floating-point sums are carried out in, and a near-tie at one token flips. Temperature 0 picks the highest-scoring token. It does not fix which token scores highest.

That distinction decides whether your last eval measured a change or measured the weather.

What makes a temperature 0 request nondeterministic?

Horace He and colleagues at Thinking Machines ran one prompt 1,000 times at temperature 0 against Qwen3-235B-A22B-Instruct-2507 on vLLM, 1,000 tokens of output each. They got 80 unique completions back, and the most common one appeared 78 times. All 1,000 were identical, word for word, up to the 102nd token. At the 103rd, 992 of them continued "Queens, New York" and 8 continued "New York City". The post is dated 10 September 2025 and ships the kernels it describes.

The mechanism is batch invariance, and ordinary kernels do not have it. Attention reduces over a dimension. So do matrix multiplication and RMSNorm. Add the same numbers in a different order in floating point and you get a slightly different number. Batch size sets that order, concurrent traffic sets batch size, and none of that traffic is yours.

The prompt is yours. The batch is not.

Can you turn determinism on?

Not on a hosted endpoint, and on Claude the knob has been taken away entirely.

Anthropic's Messages API reference now marks temperature as deprecated: "Models released after Claude Opus 4.6 do not support setting temperature. A value of 1.0 will be accepted for backwards compatibility, all other values will be rejected with a 400 error." The same parameter description still carries the older warning, that "even with temperature of 0.0, the results will not be fully deterministic." The Claude Sonnet 5.5 migration guide extends it: "Claude Sonnet 4.6 and earlier models and Claude Haiku 4.5 accept temperature, top_p, and top_k. On Claude Sonnet 5.5, a non-default value returns a 400 error. Remove them."

Where you run it	What the vendor documents	What you get
Claude API	temperature deprecated after Claude Opus 4.6, non-default values rejected with 400	No sampling control, and no determinism guarantee
OpenAI API	seed is a "best effort to sample deterministically", watched through system_fingerprint	"A small chance that responses differ even when request parameters and system_fingerprint match"
Self-hosted vLLM	VLLM_BATCH_INVARIANT=1, beta, NVIDIA compute capability 8.0 or higher	Output "independent of the batch size or the order of requests in a batch", at a documented performance cost

OpenAI's wording comes from its cookbook entry on the seed parameter, which now carries an archive notice. The vLLM wording is from the batch invariance page, which calls the feature beta.

The deterministic path has a price and Thinking Machines measured it. 1,000 sequences of Qwen3-8B on a single GPU took 26 seconds on default vLLM and 55 seconds on the first batch-invariant build, falling to 42 once they rewrote the attention kernel. Reproducibility costs roughly 60% more wall clock, and it is only for sale to whoever runs the server.

What does this do to an eval?

It widens an interval that is already wider than most teams assume. Take a 200-case eval returning 80%. The standard error of that proportion is the square root of 0.8 times 0.2 divided by 200, which is 2.8 points, so a 95% interval runs from about 74% to 86%. A four-point gain over last week's build sits inside that interval before one token of run-to-run variation is added on top. Evan Miller's Adding Error Bars to Evals sets out the formulas, including the paired comparison that recovers some of the lost power.

Measuring the spread is cheap. That is what makes skipping it hard to defend. Run the eval five times against the same build before you compare it with anything, and at list prices checked on 16 October 2026, 200 cases of 4,000 input and 600 output tokens on Claude Sonnet 5.5 come to $2.80 a run: 800,000 input tokens at $2 per million is $1.60, and 120,000 output tokens at $10 per million is $1.20. Five runs is $14.

The same arithmetic lands on two kinds of team. One has 2,000 recorded cases and a nightly job, so a four-point move shows up against its own history. The other has 40 hand-picked cases and a screenshot pasted into a Slack thread, and ships a prompt rewrite on a difference that was never there. A forty-case eval cannot see a four-point change.

What does it mean for replaying an agent?

That a transcript is a record and not a reproduction. Re-run yesterday's agent over the same inputs and it can diverge at token 103, pick a different tool and reach the same place by another route, with nothing broken anywhere. We argued the same point from the other side in How do you stop an AI agent from doing the same thing twice?: the check belongs against the world, not against the model's own account of what it did. Store the output. Do not store the recipe and expect to re-derive the output from it.

This matters most where the agent touches something outside itself. We built OFFSET, which files declarations directly to the ΑΑΔΕ portal. A system of that shape reads a rendered page and decides which field to fill, so a run that diverges at token 103 does not return a different paragraph. It clicks somewhere else.

The same discipline applies to comparing two designs. Does a multi-agent AI system beat a single agent? is the write-up of that mistake at a larger scale, where a 90.2% headline came out of a comparison whose two sides were never given the same token budget. Unequal conditions and unmeasured variance produce the same artefact: a number that moves for a reason nobody wrote down.

What to change this week

Pin the dated model snapshot in the request rather than an alias, and log the string you actually sent next to every result. Store every output. You will not be able to regenerate it. Run the eval against the same build several times before you run it against a new one, then publish the spread beside the mean. Date every number you write down, including the ones in the note that never leaves the company.

We have not measured divergence rates on a hosted frontier endpoint ourselves, and we could not find anyone who has published one. The 80-in-1,000 figure is Qwen3 served by vLLM. A managed endpoint on a different stack may be tighter or looser. We also failed to pin Google's documented position on seed to a primary page, which is why the table above has three rows rather than four.

How many times has your eval been run against the same build, and what was the spread? If that number does not exist yet, establishing it before the next model swap is the kind of work we scope.