Almost Valid JSON: How to Force a Model to Hold Shape

The failure mode is specific and everyone who has shipped a feature on top of a model knows it. Ninety seven times out of a hundred you get clean JSON. The other three times you get clean JSON wrapped in a markdown fence, or a trailing comma, or a field the schema never mentioned, or a perfectly formed object where a number arrived as the string "twelve". In a script you shrug. In a queue processing orders overnight, three percent is a pager.

There are four ways to fix this and they are not variations on one idea. They work at different layers, they fail differently, and two of them can be combined to cover what neither covers alone. What follows is each one, what it actually guarantees, and what it costs.

Why does asking for JSON in the prompt stop working?

Because nothing enforces it. A prompt is a preference expressed to a system that samples tokens probabilistically, so an instruction to emit JSON shifts the distribution without bounding it. The failure rate is low enough to pass a test suite of ten cases and high enough to matter across ten thousand.

The uncomfortable part is that failure correlates with exactly the inputs you care about. Short, ordinary inputs produce clean output. Long inputs, unusual characters, text in another language, anything that makes the model uncertain: those are the ones that come back malformed. So the cases that break are the cases where a human would have wanted the extraction to work hardest.

This is also why prompt only approaches look fine right up until they do not. Testing does not sample from the same distribution as production, and the tail is where the whole problem lives.

What does constrained decoding actually do?

It removes the possibility of invalid output rather than reducing its likelihood, by restricting which tokens the model is allowed to sample at each step to those that keep the output on a valid path through the schema.

OpenAI's guide to structured model outputs states the guarantee directly: the model will always generate responses that adhere to the supplied JSON Schema, so required keys are not omitted and invalid enum values are not invented. The documentation is explicit that this is a different thing from JSON mode, which only ensures the output is valid JSON and says nothing about whether it matches your schema. If you have been using JSON mode and still getting shape errors, that is the reason, and it is not a bug.

The open source route works the same way underneath. The Outlines library describes itself as guaranteeing structured outputs during generation directly from any model, and supports regular expressions, JSON Schema, Pydantic models and context free grammars as the constraint, across local runtimes and hosted APIs alike. The mechanism is identical: filter the token distribution to what the grammar permits.

What do you give up to get the guarantee?

A slice of JSON Schema and a slice of flexibility, both documented and both larger than people expect on first read.

The OpenAI guide lists the constraints plainly. All fields must be marked required, which means optional data has to be modelled as a nullable union instead. Every object must set additionalProperties to false. A schema may carry up to 5000 object properties in total across up to 10 levels of nesting. The combined length of property names, definition names and enum values cannot exceed 120,000 characters. The first request with any new schema pays extra latency while the API processes it, and subsequent requests with that same schema do not, which matters if you generate schemas dynamically per user rather than defining a fixed set.

There is a behavioural cost too, and it is the one nobody mentions on a pricing page. A peer reviewed study, Let Me Speak Freely, presented at the EMNLP 2024 industry track, evaluated models restricted to structured formats against the same models producing free form answers. The finding was a significant decline in reasoning under format restriction, and stricter constraints produced greater degradation. The authors call the result surprising, which it is if you assume the format is orthogonal to the thinking.

The practical response is not to abandon constraints. It is to stop asking one call to reason and format simultaneously. Let the model work through the problem in prose, then make a second, cheap, constrained call whose only job is to turn that prose into the object. You pay for two calls and get the reasoning quality of the unconstrained one with the reliability of the constrained one.

The four approaches, side by side

Each row below is a different layer of the stack, which is why the last column matters more than the second. These are not four competing products, they are four places to intervene.

Is forcing a tool call the same thing?

Close enough to be interchangeable for extraction work, and it is often the shorter path because the plumbing is already there. If the extraction in question is pulling line items off supplier invoices, the schema is only half the job and the arithmetic check is the other half.

Anthropic's overview of tool use documents the shape: you pass a tool with an input schema, and the model returns a tool use block whose arguments follow it. The default tool choice is auto, which leaves the decision to the model. Setting it to any or to a named tool removes that decision. And the guarantee has an explicit switch: adding a strict flag to a custom tool definition makes the tool calls match your schema exactly.

The trick people miss is that a tool does not have to do anything. Define one tool called record_extraction whose input schema is the object you want, force it, and ignore the fact that no function exists on your side. You have used the tool calling machinery purely as a structured output channel, which is what most extraction pipelines are doing whether or not they describe it that way.

The cost is measurable rather than theoretical. The same page publishes the token overhead of enabling tool use per model. On one current model the tool use system prompt is 286 tokens with tool choice set to auto or none and 406 tokens when a call is required, and those are added on top of your own tool names, descriptions and schemas, which are themselves counted as input tokens on every single request. For a high volume extraction job that overhead is a real line on the bill, and it is the kind of thing that goes unnoticed until someone reads the invoice properly. We went through how those numbers add up in general in the piece on why an LLM bill comes in larger than the estimate.

Why do you still need validation if the schema is enforced?

Because schema conformance and correctness are different properties, and only one of them is being guaranteed. An object that satisfies every type constraint can still be nonsense.

Consider extracting a delivery date from a customer email. Constrained decoding will give you a string matching a date format every time. It will not tell you the model read "the 3rd" as this March when the customer meant next Tuesday. Consider extracting a quantity. You will get an integer. You will not get told that the integer came from an order line the model misread.

So the layers stack rather than replace each other. Constrained decoding or a strict tool call handles shape, which removes the parse errors and the fence stripping and the whole category of problems that produce stack traces. Business validation handles sense, which removes the category that produces angry customers. Anyone who has replaced the second with the first has swapped a loud failure for a quiet one.

There is also a refusal path worth wiring in. The OpenAI documentation notes that when a model declines a request for safety reasons the response carries a dedicated refusal field, because a refusal will not conform to your schema. Code that assumes the schema always applies will throw on a case that is not actually an error.

What should you build first?

Start at the layer that matches the consequence of being wrong, not at the most sophisticated one.

• Internal tooling, a human reads the output. Prompt plus a parse. If it breaks, the person sees it break. Adding machinery here is cost without benefit.
• Anything that writes to a database unattended. Strict tool call or constrained decoding, plus validation on the values. This is the default for production work and the point where the extra call stops being optional.
• Anything that requires reasoning before the structure. Two calls. Reason in prose, format under constraint. The study above is the reason, and the second call is usually the cheapest one in the pipeline.
• High volume, simple extraction. Constrained decoding on a fixed schema, so you pay the schema processing latency once rather than per request.
• A local or self hosted model. A grammar based library, since the hosted structured output feature is not available to you and prompting a smaller model for JSON is where this problem is at its worst.

The wrong move is the one that feels most responsible, which is to add every layer at once on day one. Retry logic on top of constrained decoding on top of a validation schema means three places for a bug and three things to reason about when output goes missing. Add the layer whose failure you have actually observed.

What about a model you host yourself?

The hosted structured output feature does not exist for you, so the guarantee has to come from the runtime. This is where the grammar libraries stop being an alternative and become the only option.

Outlines lists the surfaces it plugs into: transformers and llama.cpp for local models, vLLM and Ollama for servers, and hosted APIs alongside them. The same code expresses the constraint in all of those, which is the useful property, because it means the decision about where a model runs stops being coupled to the decision about how you enforce shape. A pipeline written against a grammar can move from a laptop to a server to an API without the extraction layer changing.

Self hosting also raises the stakes on this whole topic rather than lowering them. Smaller open weight models are noticeably worse at holding a format from instructions alone, so the gap between prompting and constraining is widest exactly where people reach for a small model to save money. Anyone planning to run inference themselves should treat grammar constrained decoding as part of the setup rather than an optimisation, and our walkthrough of what your hardware can realistically run locally covers the rest of that decision.

What this changes about how you write the schema

More than people expect, because a constrained schema is a specification the model reads rather than a check you run afterwards.

Field names carry meaning to the model, so naming a field ship_by_date rather than d2 measurably changes what lands in it. Descriptions on fields are instructions, not documentation. Enums are the strongest tool available in this whole area, because an enum of five permitted values makes four hundred wrong answers unreachable, which no amount of prompting achieves. And since every field has to be required, the honest way to express optionality is a union with null, which forces you to decide what missing actually means for each field rather than discovering it later.

That last point is the quiet benefit of the restriction. Being forced to declare every field and forbid every extra one produces a tighter data model than the one most people write freehand. The constraint is doing design work.

If you are wiring model calls into a real product rather than a script, the surrounding decisions matter as much as this one, and we walked through what the layer around those calls has to own in the piece on what an AI orchestration layer has to handle once you are several calls deep. For connecting a model to your own systems through a standard interface rather than bespoke glue, our MCP server page covers the protocol side, and the walkthrough on setting up an MCP server in practice covers the build.

The short version

If output shape has ever broken your pipeline, prompting is not the fix and a better prompt is not the fix either. Move the guarantee from the instruction into the decoder, keep validation for meaning, and stop asking one call to think and to format at the same time. The three of those together cover every failure described here, and none of them is more than an afternoon of work.