Does forcing JSON output make an AI model worse at reasoning?

Not by itself. Schema-constrained output costs accuracy when a model has no headroom left for the task, and costs nothing measurable when it has headroom to spare. The variable is capacity rather than format. What fixes it is the order you ask for things in.

Every number below was checked by Unity Horizon on 22 September 2026 against the papers and the vendor documentation linked in place.

Where does the claim that JSON hurts reasoning come from?

One paper. Tam, Wu, Tsai, Lin, Lee and Chen published Let Me Speak Freely? at the EMNLP 2024 Industry Track, and its abstract says "Surprisingly, we observe a significant decline in LLMs reasoning abilities under format restrictions." That sentence has carried two years of posts titled some version of "JSON mode makes your model dumber".

The .txt team replied with Say What You Mean and named five defects. The one that decides the result: the structured and unstructured runs did not get the same prompt, and the JSON prompt never said which schema to use. Their replication on Llama-3-8B-instruct put structured generation ahead on all three reasoning tasks.

Task, Llama-3-8B-instruct	Unstructured	Structured
GSM8K	0.77	0.78
Last Letter	0.73	0.77
Shuffle Object	0.41	0.44

The paper measured a prompt. Everyone quoted it as a property of JSON.

Does the penalty show up on current models?

On hard tasks it does, and the spread between models runs to 36 points. Hengxin Fan's Capacity, Not Format, posted 8 June 2026, ran four models over five benchmarks with information-matched prose controls and reports 0% parse failures on successfully generated responses.

Benchmark and model	Chain of thought	JSON	Change
MATH-Hard, Claude Sonnet 4.6	89.3%	88.7%	none detected
MATH-Hard, Claude Haiku 4.5	88.7%	52.5%	down 36.2 points
MATH-Hard, GPT-4o-mini	62.3%	34.3%	down 28.0 points
AIME, Claude Opus 4.7	96.2%	91.0%	down 5.3 points

Now the part that the headline version of this argument always drops. On GSM8K, every model in the study scored between 87% and 97% under every condition, and no format produced a statistically significant change for any of them. The penalty needs a task the model finds hard before it exists at all. Run your format comparison on grade-school arithmetic and you will measure nothing, then ship the setting into a workload where it costs thirty points.

Why would a schema cost accuracy at all?

Constrained decoding compiles your JSON Schema into a grammar, then masks every token that would break it and renormalises what is left. So from the first token, the model is emitting output that has to sit inside the structure. The tokens it would have spent working the problem out are tokens it now spends filling fields.

Fan rules out the comfortable explanation. Under extended token budgets that eliminated truncation, GPT-4o-mini still dropped 28.0 points. The model is not running out of room. It is running out of attention while holding a grammar state open across thousands of tokens, and that is a different problem with a different fix.

The ablation is the proof and it is also the fix. Let the model reason freely and format afterwards, and Haiku goes from 54.3% to 82.0%, recovering 79.5% of the gap to chain of thought. A separate March 2026 preprint, Draft-Conditioned Constrained Decoding, reached the same place from the other direction: generate an unconstrained draft, then decode under the grammar conditioned on that draft, for gains of 5.9 to 14.1 absolute points across models from 1B to 14B.

Does this apply to tool calls in an agent?

Yes, and harder. A forced tool call is the most constrained output an agent produces. Anthropic's strict tool use sets strict: true on a tool definition and constrains sampling to schema-valid inputs. The page calls it grammar-constrained sampling. What it guarantees is stated plainly: the tool input follows your input_schema and the tool name is valid. Types, and not judgement.

Fan's production ladder, GPT-4o-mini on MATH-Hard, puts a number on each rung.

How the output was requested	Accuracy
Free-form chain of thought	60%
API JSON mode	51%
JSON asked for in the instruction	34%
Forced function calling	10%

Forced function calling is the mode where the model emits arguments and nothing else. That is the shape of a request that pins tool_choice to a named tool, and the shape of most wrappers written to get a structured result out of a model in one call.

This is the second accuracy effect living in the same code path. In how many tools an AI agent can have before it picks the wrong one the failure was selection: past 30 to 50 tools the model reaches for the wrong one. The failure here arrives one step later, after the model has picked correctly and has to commit to arguments before it has worked out what they should be.

One boundary worth knowing. The computer use and browser use toolset entries, computer_toolset_20260801 and browser_toolset_20260801, do not accept strict: true, and a request that sets it on either entry is rejected. We built OFFSET, which files declarations directly to the ΑΑΔΕ portal, and that work runs down the computer use path, where the grammar was never on offer in the first place.

What does a schema buy, and where does it stop?

Validity, at every model size. Akash Chavan's Constrained Decoding Eliminates Structural Failures in Small LLMs, posted 20 September 2026, ran five models from 0.6B to 4B parameters over 14 structured-output tasks. Schema validity went from a native range of 78.6% to 92.9% up to 100% for all five under both Outlines and XGrammar. Content accuracy went up too on the smallest models, with Llama 3.2-1B moving from 0.771 to 0.986 under XGrammar.

Then the sentence that settles the whole argument for anyone building on top of this: "Schema conformance is necessary but not sufficient for semantic correctness; CD's reach ends exactly where schema conformance ends." Type coercion failures are fully rescued by the grammar. Instruction-semantic failures, and the paper names multi-step function calling as one, are not.

The parser stopped failing. The answer kept being wrong.

Who pays for this, and who never notices?

A team on a frontier model pulling fields out of a document pays nothing. Turn strict on everywhere, delete the retry loop, go home. A team that moved to a smaller model to cut the inference bill and switched strict on in the same sprint pays twice, and the two decisions were made in different meetings by people who were both right on their own terms.

The same line runs through tool providers. A tool whose arguments are looked up or copied is easy to drive under a grammar. A tool that takes a nested object and expects the model to decide which branch applies has moved the reasoning inside the schema. That is the one place in the request where the model has the least room to do it. No design review catches this, because a schema looks like typing.

What does reasoning first and formatting second cost?

Six tenths of a cent a turn, at Claude Opus 5 list prices of $5 per million input tokens, $0.50 per million cache read tokens and $25 per million output tokens. The turn below has a 4,000 token prefix already in a 5 minute cache and about 900 tokens of output.

Line, Claude Opus 5	One pass, schema from token one	Two passes, reason then format
First call, input	$0.020	$0.020
First call, output	$0.023 (920 tokens)	$0.020 (800 tokens)
Second call, input	n/a	$0.006 (4,000 cached, 800 new)
Second call, output	n/a	$0.003 (120 tokens)
Per turn	$0.043	$0.049

On Fan's numbers that same ordering change is worth 27.7 points on Haiku 4.5 and nothing at all on Sonnet 4.6, so what the six tenths of a cent buys is insurance against being on the wrong side of that split. You also pay a second round trip. That matters on a chat surface and it does not matter in a batch job.

We have not run this ablation on our own agents, so read the recommendation as arithmetic over other people's measurements rather than as our number. Three of the four papers are preprints, and the strongest of them is a single author whose model line-up mixes generations. The direction has been reproduced independently. The size has not.

Which step in your agent emits its answer straight into a schema, and has anyone run that step once with the schema taken off? If nobody has, that is the measurement we would start with.