Most of us here have shipped something that calls a model and then spends half its code cleaning up whatever came back. This is the writeup I wish I had the first time I wired an LLM into a real pipeline, so I am leaving it here for anyone about to do the same.
The Three Workarounds We All Wrote First
Prompt engineering: tell the model to respond in JSON with these exact fields, and hope. That holds roughly 85 to 95 percent of the time depending on the model, which sounds fine until you are handling real traffic and it fails a few hundred times a day.
Post processing: regex parsers, JSON repair libraries, retry loops. Adds latency and a whole class of bugs that have nothing to do with your actual business logic.
Function calling: works well for invoking a tool, semantically awkward when all you want is data back rather than a call made.
What Constrained Decoding Actually Does
At each generation step the model produces a probability distribution over its entire vocabulary. Constrained decoding masks out every token that would take the output off a valid path through your schema, before sampling happens.
If your schema says sentiment must be one of positive, negative or neutral, the model cannot emit "somewhat positive". Those tokens are not available to it. That is enforced in the inference stack rather than in your prompt, which is why the guarantee is absolute instead of probabilistic.
JSON Mode Is Not Schema Enforcement
This one trips people up constantly. JSON mode guarantees the response parses. It does not guarantee the response has your fields. You can ask for a sentiment and a confidence score and get back {"answer": "yes"}, which is perfectly valid JSON and useless to your code.
Schema enforcement guarantees conformance to the contract: your fields, your types, your enums. If you are shipping to production, that is the one you want.
What It Still Does Not Give You
The schema promises that price is a number. It does not promise the number is positive, or that the email string is a real address. Schema enforcement removes structural errors and leaves semantic ones, so keep a validation layer such as Pydantic sitting on top of it.
Worth knowing before you design around it: most providers cap nesting around five levels, strict mode wants every property listed in the required array, and optional fields get expressed as anyOf with a null type.
The full breakdown, including provider by provider support and the migration path off JSON mode, is here: LLM structured output
If you have hit a case where schema enforcement made things worse rather than better, I would like to hear it. The failure modes are far less documented than the happy path.