Is a two-point gap between two AI models on a benchmark real?
Unity Horizon5 min read
Usually not. GPQA Diamond has 198 questions, so two models near 94% need a gap of about 6.7 points before one run can separate them. Checked 23 October 2026.
Usually not. GPQA Diamond is 198 questions. Two models both scoring near 94% need a gap of roughly 6.7 points before one run of that benchmark can separate them, and the gaps printed in the comparison tables are under two.
The mechanism is the denominator. A benchmark score is a sample mean over a few hundred questions, and the error on that mean is set by how many questions there are, not by how large the model answering them was.
How big does a benchmark gap have to be before it counts?
Big enough to clear the noise in the question set it was measured on. To detect a difference at 80% power and 5% significance the gap has to beat 2.8 times the standard error of the difference, which for two scores near p over the same n questions is the square root of 2p(1-p)/n. The table below is that formula and nothing else, so you can redo it in a line of Python.
| Questions in the eval | Both near 50% | Both near 70% | Both near 90% |
|---|---|---|---|
| 30 | 36.2 points | 33.1 points | 21.7 points |
| 100 | 19.8 | 18.2 | 11.9 |
| 198 | 14.1 | 12.9 | 8.4 |
| 500 | 8.9 | 8.1 | 5.3 |
| 1,000 | 6.3 | 5.7 | 3.8 |
Every row there is a benchmark somebody quotes. GPQA Diamond is 198 questions, which its authors call their highest quality subset. SWE-bench Verified is "a human-validated subset of 500 SWE-bench instances". The retrieval study we wrote about three days ago ran on 116 questions. Checked by Unity Horizon on 23 October 2026.
Why does a leaderboard show a winner when the numbers do not?
Because a ranking is a sort, and a sort returns an order whether or not the differences inside it are real. Vals compares Claude Opus 5 with GPT-5.6 Sol and puts Opus 5 ahead on 28 of the 34 benchmarks both models share. The same page says "This is not a pairwise statistical significance test", and reports that "Reported ±1 standard-error ranges overlap on 11 of 28 shared benchmarks with comparable uncertainty data". Twenty-eight rows carry error bars and eleven of them are ties.
GPQA Diamond is one of the eleven. The two scores are 93.43% with a standard error of 1.24 and 95.20% with 1.07, and combining those gives an error of 1.64 on the difference between them. So a 1.77-point gap is 1.1 standard errors. The gap would have to reach about 4.6 points before a single run could call it.
The overlap rule is the generous version of the test, too. Two single-SE bars that barely fail to touch sit roughly 1.4 standard errors apart, which puts the p value near 0.16, well inside the range where a careful reader would shrug. The real count of ties on that page is higher than eleven.
Does running the eval more than once fix it?
Partly. Adding Error Bars to Evals, Evan Miller, November 2024, splits the variance of a score in two: how much the questions differ in difficulty, and how much one model's answer to one question moves between runs. Only the second half shrinks when you answer each question K times.
His worked example takes 198 questions to a minimum detectable effect of 13.2% at one run per question. Ten runs per question brings that to 7.5%. Carry on to infinity and it settles at 6.6%, because the questions are the floor. His recommendation is blunt: "new evals should contain at least 1,000 questions in order to have good signaling ability".
Repeats are bought with tokens. Questions are bought with people. Ten runs over 198 questions is 1,980 calls and no new writing, while roughly 600 fresh questions reach the same 7.5% in 600 calls and a fortnight of somebody's attention.
Repeats do anything at all only because the same prompt does not return the same answer. A thousand identical calls at temperature 0 came back as 80 distinct completions.
Why are the error bars on a small eval too narrow?
Two reasons, and both of them flatter the result. The ICML 2025 position paper Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints, by Sam Bowyer, Laurence Aitchison and Desi Ivanova, simulated 80,000 datasets and found a nominal 95% textbook interval covering 92.5% of the time at N=100.
It gets worse at the edges. Where every answer is right or every answer is wrong the standard error is zero, so the interval has no width at all, and that is the state most internal evals reach on their easy half. The authors recommend the Wilson interval or a Beta-Binomial posterior. The first of those ships in SciPy.
The other reason is clustering. Questions that arrive in groups are not independent draws: five questions about one passage, or one question translated into nine languages. Table 4 of Miller's paper puts the clustered standard error on DROP at 1.34 against a naive 0.44, a factor of 3.05, and MGSM at 1.88 times. Ten paraphrases of one support ticket are one question wearing ten hats.
Who gains from a leaderboard nobody tests, and who pays?
Everyone in the table gains a little, which is why nothing changes. A sort with no significance test produces a winner on every row, and each model in a comparison leads somewhere, so each of them ends up with a row worth quoting. Teams whose real effect is large gain honestly: the 36.2-point drop in Does forcing JSON output make an AI model worse at reasoning? needs no statistics behind it, and a gap that size is visible on forty questions.
The bill lands on the team with a forty-case eval and a migration to decide. They run the new model and read four points. Three weeks then go on tuning prompts against a difference their eval was never able to see. It lands again on the buyer who copies a 1.77-point gap into a procurement document as though it were a ranking.
What would we check first?
Count the questions in the eval you already have. Then count the ones where the two models answered differently, because that second number is what the comparison actually runs on. Eight disagreements split five to three is nothing, whatever the two headline percentages say.
Then stop publishing a bare percentage. Publish the count and the interval instead. 34 of 40 is a Wilson interval of 70.9% to 92.9% at 95%, and that range is the finding.
We are in this too. Our own retrieval piece reported an 11.3-point average gap with grep ahead in all ten rows. Ten out of ten is a p value of 0.002 under a sign test if the rows are independent, and they are not, because every row was scored on the same 116 questions. We published the gap and no interval around it.
The last time your team changed models, how many questions decided it, and on how many of those did the two models disagree? If nobody counted, that is where we would start.
Common questions
How many questions does an AI eval need to compare two models?
About 1,000 if you want to resolve a three-point difference, which is the recommendation in Evan Miller's Adding Error Bars to Evals. At 198 questions scored once, the smallest difference a comparison can reliably detect is around 13 points when both models sit mid-range, and under 7 points when both sit near 94%.
Is a 2-point difference between two AI models on a benchmark significant?
Rarely. On GPQA Diamond, which is 198 questions, published scores of 93.43% and 95.20% carry standard errors of 1.24 and 1.07. The standard error of the difference is 1.64, so the 1.77-point gap is 1.1 standard errors and about 4.6 points would be needed at 80% power. Checked 23 October 2026.
Does running an eval multiple times make the result more reliable?
Yes, up to a floor. Answering each of 198 questions ten times instead of once moves the minimum detectable effect from 13.2% to 7.5% in Evan Miller's worked example. Past that, repeats stop paying, because the variance that remains comes from the questions differing in difficulty and only new questions reduce it.
How should an AI eval score be reported?
As a count with an interval rather than a bare percentage. 34 of 40 is a Wilson interval of 70.9% to 92.9% at 95%. Avoid the textbook normal interval on small sets: it covers 92.5% of the time at N=100 and collapses to zero width when every answer is right or every answer is wrong.