Do LLM judges really agree with humans 80% of the time?

No. The 80% is raw agreement and it is not corrected for chance. Correct it, and 21 judges from nine providers land between 0.376 and 0.511 on MT-Bench, which is fair to moderate agreement rather than parity with a person.

The mechanism is that exact match counts matching verdicts and never asks how many of them a guesser would have matched. Put a rater in front of a two-way preference task, let it read nothing and pick at random, and it agrees with your humans about half the time on the arithmetic alone. Kappa subtracts that. Roughly half the 80% survives the subtraction.

Where does the 80% number come from?

One paper, in 2023, copied since into vendor documentation without the metric attached. Zheng and twelve co-authors wrote that "strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans", in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena at NeurIPS 2023. The sentence is true about what they measured.

What travels is the number without the metric. MLflow's page has judges "achieving over 80% agreement with human evaluators and eliminating reviewer inconsistency". LangChain's has "strong LLM judges reach 80% agreement with human evaluators, which is roughly the level of agreement humans reach with each other", and names MT-Bench as the source. Neither page says the words exact match. Checked by Unity Horizon on 18 September 2026.

Why does chance correction take 38 points off?

Because kappa subtracts the agreement two raters would reach by guessing, given how often each verdict gets used at all. A judge that answers "A" seven times in ten will agree with a human who also leans towards A, for reasons that have nothing to do with the answers.

Somebody finally did the arithmetic at scale. Norman, Rivera and Hughes at the UC Berkeley School of Information put 21 judges from nine providers through three benchmarks, three protocols and 118 runs, roughly 541,000 individual judgments, and published Reliability without Validity on 17 June 2026. Their finding: "Every judge in our study exhibits substantial kappa deflation on MT-Bench: exact match overstates chance-corrected agreement by between 33.8 and 41.3 percentage points across the 21 models, with a cohort mean of 38.6 pp."

Exact match on MT-Bench ran from 0.788 to 0.851 across those 21 models. Here is where each benchmark lands once the guessing is taken out, read against the Landis and Koch scale that has been the reference for kappa since 1977.

Benchmark	Cohen's kappa across 21 judges	Spread	Best judge reads as
MT-Bench, open-ended preference	0.376 to 0.511	13.5 points	moderate
JudgeBench, correctness	0.271 to 0.875	60.4 points	almost perfect
RewardBench, correctness	0.616 to 0.898	28.1 points	almost perfect

Read the spread column. On JudgeBench the gap between the best judge and the worst is 60.4 points of kappa, and the authors report that the rankings themselves move between benchmarks. The judge you picked because it topped somebody's leaderboard is not the judge that tops the next one.

Can a judge be consistent and biased at the same time?

Yes, and this is the part that should bother anyone who bought the phrase "eliminating reviewer inconsistency". Qwen 3 8B scored 0.992 on test-retest reliability and 0.192 on position bias. Gemini 2.5 Flash scored 0.988 and 0.125. Ask either one the same question ten times and it gives the same verdict ten times. Swap the order the two answers are pasted in, and it changes its mind.

Position flip rates on MT-Bench ran from 0.035 for Gemini 3.1 Pro to 0.189 for Qwen 3 8B. The authors name it the consistency-bias paradox and turn it into an explicit gate: once test-retest passes 0.95, confirm position bias is under 0.10, because "high stability with high bias is a failure mode".

The judge repeats itself perfectly. It also changes its mind when you swap the columns.

What do you run before you trust a judge?

Their Minimum Viable Validation Protocol is five steps, and none of them needs tooling you do not already have. Kappa or Krippendorff's alpha becomes the headline number and exact match drops to a footnote. Every comparison runs twice, AB and BA, and the rate at which the verdict flips between the two orders is the position bias. Three runs minimum, temperature 0, response caching off, because a cached verdict is not a second opinion. Two benchmarks at least, one preference-style and one correctness-style, since the rankings move between them. Then the gate: a test-retest score above 0.95 means nothing until the position bias under it has been read.

Step one is the awkward one. Kappa has to be computed against human labels, and the judge was hired so that nobody would have to sit and produce human labels. A few hundred of them, bought once, tell you what your judge is worth on your own rubric. You cannot skip the labels. You can only decide how many you buy.

Who gains from the 80%, and who pays for it?

Teams grading something with a right answer gain, and the study says so: RewardBench and JudgeBench are correctness-style sets, and that is where the top kappas sit. So does any vendor selling evaluation by the judgment, because raw agreement is the flattering metric and nobody has been asking which one the dashboard prints.

The bill lands on the small team grading open-ended text: support replies, summaries, sales drafts and concierge answers. That is the shape of MT-Bench, where kappa was lowest and the ceiling sat at 0.511. It is also the work nobody can afford to grade by hand, which is why the judge got hired there first. We built askHermis, which answers 85% of guest queries instantly, and whether a guest answer was any good is a preference judgement rather than a unit test.

There is a second effect underneath, and here we are inferring rather than measuring. The judge is a model, so it carries the same language gap the agent does. Claude Haiku 4.5 scores 52.7% of its English result in Yoruba. A judge grading Yoruba replies is scoring in a language it is weaker in, using a metric that already flatters it.

The Berkeley study is three English preference and correctness benchmarks, run by academics on public data. It is not your rubric on your transcripts, and we have not published a kappa on ours either, so take 38.6 points as the size of the correction on published benchmarks rather than a number to subtract from your own dashboard.

Which number does your eval dashboard print, exact match or kappa, and has anyone run the comparisons in both orders? If nobody has, that is where we would start.