How much worse is an AI agent in languages other than English?

Between 1.8 and 47.3 percentage points worse, depending on the language and the model tier you bought. Anthropic publishes the table. Claude Sonnet 4.5 scores 98.2% of its English result in Spanish and 79.7% in Yoruba; Claude Haiku 4.5 scores 96.4% and 52.7%.

The mechanism is that the language count and the quality come from two different places. A model accepts any language that fits in Unicode, which is an encoding fact and costs the vendor nothing to claim. It is good at the languages there was a lot of text in. The language list is a routing table.

What does "supports 38 languages" mean on a vendor page?

It means the text goes in and something comes back in the same script. Anthropic's multilingual support page says so in its own words: "Claude is capable in many languages beyond those benchmarked in the following table. Test with any languages relevant to your specific use cases." Fourteen languages are measured. The rest are your problem.

We built askHermis, which supports 38+ languages and answers 85% of guest queries instantly. What happens to the other 15% is the part that took the design work. The 38 describes who is allowed to write in. It does not describe what comes back when a guest sends a message in Bengali at twenty to midnight and the night manager reads Greek and English.

Which languages lose the most when you move to a cheaper model?

Anthropic's table, with extended thinking on, scores expressed as a percentage of the same model's English performance, with the third column subtracted by us. Checked by Unity Horizon on 14 September 2026.

Language	Sonnet 4.5	Haiku 4.5	Difference between the two gaps to English (points)
English	100%	100%	0.0
Spanish	98.2%	96.4%	1.8
French	97.5%	95.7%	1.8
Italian	97.9%	96.0%	1.9
German	97.0%	94.3%	2.7
Japanese	96.8%	93.5%	3.3
Hindi	96.7%	92.4%	4.3
Arabic	97.2%	92.5%	4.7
Bengali	95.4%	90.4%	5.0
Swahili	91.1%	78.3%	12.8
Yoruba	79.7%	52.7%	27.0

Read the last column, because it is the one nobody prints. One procurement decision, taken once, in a meeting about the invoice. In Spanish, Haiku 4.5 sits 1.8 points further from its own English score than Sonnet 4.5 does. In Yoruba it sits 27.0 points further. Each column is measured against its own model's English result, so the difference shows where the cheaper model falls away from English faster. It does not give the absolute drop a customer sees after the switch. The discount is priced in dollars. The loss is priced in the language your customer writes in.

Two vendors, the same ordering. The underlying set is MMLU translated into fourteen languages by professional human translators, published in OpenAI's simple-evals repository, and it holds down that table too: o3-high scores 0.912 in Italian and 0.780 in Yoruba, while gpt-4.1-nano scores 0.734 and 0.455.

Does a message in another language cost more to run?

It usually costs more tokens for the same meaning. Petrov, La Malfa, Torr and Bibi showed in Language Model Tokenizers Introduce Unfairness Between Languages that the same text translated into different languages can differ in tokenized length "up to 15 times in some cases", and that this hits cost, latency and how much fits in the context window. A 2026 measurement across 25 European languages on six published tokenizers puts English at 1.23 tokens per word and Greek and Maltese at about 3.1. Ukrainian comes in at 2.66 against 2.25 for Polish, which the author attributes to pretraining data volume rather than to the script.

Arithmetic over list prices, checked 11 September 2026. Anthropic's own support costing puts a customer conversation at roughly 3,700 tokens and 10,000 of them at about $37 on Haiku 4.5, per the pricing page. Assume a third of those tokens are the customer's own text and the rest is your English system prompt and history. Arriving in Greek, that third goes from about 1,233 tokens to about 3,100, and the same 10,000 conversations cost around $56.

The fertility figures come from the Llama, Gemma, GPT-4o, DeepSeek and Qwen tokenizers rather than from Claude's, and Anthropic's pricing page notes that Claude 4.7 and later models moved to a newer tokenizer producing "approximately 30% more tokens for the same text" than the one before it. So the ratio is a shape rather than a coefficient. Count your own traffic, through your own tokenizer.

How do you test a chatbot in a language nobody on your team speaks?

Count the traffic first. Take every language above one percent of your inbound messages and ignore the rest of the list, because a count of 38 says nothing about which of them your customers write in.

Then build the eval set out of your own transcripts and have a person translate it. That is what the serious benchmarks do and it is why they are rare: the European Commission's DG Translation released EU MMLU in July 2026 using nearly 250 students from 21 universities in the European Master's in Translation network to translate and revise just over 1,000 questions into 16 EU languages. Machine translation is the cheap substitute and it carries the original language inside it.

That last part is measurable. Global MMLU, which covers 42 languages, found that 28% of MMLU questions need culturally sensitive knowledge and that 84.9% of the geography questions are about North America or Europe, and that model rankings change depending on whether you score the full set or only the culturally sensitive subset. A translated English benchmark tests a translated English world. Your guests ask about the parking, the late checkout, the ferry timetable and whether the tap water is drinkable.

Set the handover threshold per language as well. The confidence score that should mean "answer it" in Spanish is a different number in Swahili, and a single global threshold hands that whole difference to the customer.

Who gains from a single language count, and who pays for it?

Companies whose customers write Spanish, Portuguese, Italian or Indonesian gain. They can move to the cheap tier and book the saving against a gap to English about two points wider. So can any vendor selling on a count, because the count is free to raise and nobody audits it.

The bill lands on the operator whose inbox is a third Swahili, who cut the model spend by 67% on a Tuesday and read about the result in a review three weeks later. The same asymmetry runs through the tokenizer. Ukrainian costs roughly 18% more tokens per word than Polish, for reasons that measurement puts in the volume of pretraining text rather than in the alphabet, so the languages with the least text behind them are billed most for the worst answers.

askHermis answers 85% of guest queries instantly, and we have not published a per-language breakdown of it. Everything above is the vendors' measurement and the benchmarks' measurement rather than ours, and the tokenizer arithmetic is an illustration built on somebody else's tokenizers.

Which language is second in your support inbox, and what does your agent score in it? If nobody has counted, counting is where we would start.