Why Your LLM Still Returns Broken JSON

Leader 2 8 92
calendar_today agoschedule2 min read

Most of us here have shipped something that calls a model and then spends half its code cleaning up whatever came back. This is the writeup I wish I had the first time I wired an LLM into a real pipeline, so I am leaving it here for anyone about to do the same.

The Three Workarounds We All Wrote First

Prompt engineering: tell the model to respond in JSON with these exact fields, and hope. That holds roughly 85 to 95 percent of the time depending on the model, which sounds fine until you are handling real traffic and it fails a few hundred times a day.

Post processing: regex parsers, JSON repair libraries, retry loops. Adds latency and a whole class of bugs that have nothing to do with your actual business logic.

Function calling: works well for invoking a tool, semantically awkward when all you want is data back rather than a call made.

What Constrained Decoding Actually Does

At each generation step the model produces a probability distribution over its entire vocabulary. Constrained decoding masks out every token that would take the output off a valid path through your schema, before sampling happens.

If your schema says sentiment must be one of positive, negative or neutral, the model cannot emit "somewhat positive". Those tokens are not available to it. That is enforced in the inference stack rather than in your prompt, which is why the guarantee is absolute instead of probabilistic.

JSON Mode Is Not Schema Enforcement

This one trips people up constantly. JSON mode guarantees the response parses. It does not guarantee the response has your fields. You can ask for a sentiment and a confidence score and get back {"answer": "yes"}, which is perfectly valid JSON and useless to your code.

Schema enforcement guarantees conformance to the contract: your fields, your types, your enums. If you are shipping to production, that is the one you want.

What It Still Does Not Give You

The schema promises that price is a number. It does not promise the number is positive, or that the email string is a real address. Schema enforcement removes structural errors and leaves semantic ones, so keep a validation layer such as Pydantic sitting on top of it.

Worth knowing before you design around it: most providers cap nesting around five levels, strict mode wants every property listed in the required array, and optional fields get expressed as anyOf with a null type.

The full breakdown, including provider by provider support and the migration path off JSON mode, is here: LLM structured output

If you have hit a case where schema enforcement made things worse rather than better, I would like to hear it. The failure modes are far less documented than the happy path.

🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

The Zero-Net-Loss Fleet & The Mercenary Squad: A Live AI Economy

DEVPlank - Aug 4

The Privacy Gap: Why sending financial ledgers to OpenAI is broken

Pocket Portfolio - Feb 23

Sovereign Intelligence: The Complete 25,000 Word Blueprint (Download)

Pocket Portfolio - Apr 1

Merancang Backend Bisnis ISP: API Pelanggan, Paket Internet, Invoice, dan Tiket Support

Masbadar - Mar 13

Dashboard Operasional Armada Rental Mobil dengan Python + FastAPI

Masbadar - Mar 12
chevron_left
3.8k Points103 Badges
United Statest.co/5LlztlB5C5
91Posts
17Comments
16Connections
Our AI Apps are a self expanding AI SaaS ecosystem used to create the custom web application of your... Show more

Related Jobs

View all jobs →

Commenters (This Week)

3 comments
1 comment
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!