You asked the model for JSON. You even wrote "respond ONLY with valid JSON, no other text" in capital letters. And it mostly works — until the one call in fifty where the model wraps the object in `json fences, prepends "Sure! Here's the data you requested:", invents a field you never asked for, or drops a closing brace. Your json.loads() throws, your pipeline halts, and you reach for another try/except and a regex to strip fences.
Stop. That whack-a-mole is a symptom of solving the problem at the wrong layer. Getting reliable structured data out of an LLM is a solved problem in 2026 — but only if you understand why free-form prompting fails and which of the four real mechanisms actually guarantees what you need. Let's fix it properly.
Why prompting for JSON is fundamentally fragile
Here's the core issue: an LLM generates text one token at a time, sampling from a probability distribution over its entire vocabulary at every step. When you ask for JSON in the prompt, you've made valid JSON likely — but nothing in the generation process makes it mandatory. At any token, there's a nonzero probability the model emits a backtick, a friendly preamble, a markdown fence, or a plausible-but-unrequested key. Over enough calls, "unlikely" becomes "happens in production every day."
This is why the usual defenses are so brittle:
- "Respond only with JSON" in the prompt reduces the failure rate but never eliminates it. It's a suggestion, not a constraint.
- Regex to strip fences and preamble is a losing arms race against the infinite creativity of a model's off-script behavior. You'll patch fences, then hit trailing commentary, then a comment inside the JSON, then a stray Unicode quote.
try/except json.loads() with a retry works until the model returns syntactically valid JSON that's semantically wrong — right shape, missing your required field, wrong types. It parses cleanly and corrupts your data downstream.
The mistake underneath all three is treating the output as something to sanitize after the fact. The fix is to constrain the generation so invalid output can't be produced in the first place. There are four ways to do that, in increasing order of guarantee.
Approach 1 — JSON mode (valid syntax, not your schema)
Most major providers offer a JSON mode: a flag that guarantees the output is syntactically valid JSON. Turn it on and you'll never again get markdown fences or a chatty preamble — the response parses every time.
The catch, and it's a big one: JSON mode guarantees valid JSON, not valid your JSON. The model can still return {}, omit your required amount field, put a string where you expected a number, or invent extra keys. You've eliminated parse errors but not schema errors. JSON mode is a floor, not a ceiling — useful, but insufficient on its own for anything that feeds a typed system downstream.
Tool calling (also called function calling) is how most production systems have gotten structured data since it shipped, and it's still the default reach-for. You describe a function with a JSON Schema for its parameters; the model, instead of writing prose, returns a structured call to that function with arguments matching your schema.
# Anthropic tool use — the tool's input_schema defines the shape you want back.
import anthropic
client = anthropic.Anthropic()
tools = [{
"name": "record_expense",
"description": "Record a structured expense extracted from a receipt.",
"input_schema": {
"type": "object",
"properties": {
"amount": {"type": "number"},
"currency": {"type": "string"},
"category": {"type": "string", "enum": ["travel", "food", "software", "other"]},
},
"required": ["amount", "currency", "category"],
},
}]
msg = client.messages.create(
model="claude-sonnet-5",
max_tokens=512,
tools=tools,
tool_choice={"type": "tool", "name": "record_expense"}, # force the tool
messages=[{"role": "user", "content": "Coffee with a client, 38 lei."}],
)
# The structured arguments arrive as a dict, not a string to parse.
tool_use = next(b for b in msg.content if b.type == "tool_use")
print(tool_use.input) # {'amount': 38, 'currency': 'RON', 'category': 'food'}
Two things make this robust. First, tool_choice forces the model to call the tool rather than deciding whether to — no "here's the answer in prose" escape hatch. Second, the arguments arrive as structured data keyed by your schema, not a blob of text you have to rescue. Tool calling is also the exact same mechanism agents use to act on the world, which is why getting it reliable is foundational to any autonomous system — the heart of a dedicated course on building and automating AI agents.
The honest limitation: historically, tool-calling arguments were strongly encouraged to match the schema but not hard-guaranteed by the decoder on every provider. That gap is what the next approach closes.
Approach 3 — Structured Outputs with strict schema enforcement
The current best answer is Structured Outputs (OpenAI's term; other stacks call it schema-constrained or grammar-constrained decoding). You supply a JSON Schema and the provider guarantees the output conforms to it — not by validating afterward, but by constraining the decoder so tokens that would violate the schema are never sampled in the first place.
The technique underneath is constrained decoding: at each generation step, the sampler is restricted to only those tokens that keep the output on a valid path through your schema's grammar. If the schema says the next thing must be a number, the model literally cannot emit a letter. Invalid output isn't cleaned up — it's made unreachable.
# OpenAI Structured Outputs with a Pydantic schema — parsed straight into a typed object.
from openai import OpenAI
from pydantic import BaseModel
from enum import Enum
class Category(str, Enum):
travel = "travel"
food = "food"
software = "software"
other = "other"
class Expense(BaseModel):
amount: float
currency: str
category: Category
client = OpenAI()
resp = client.responses.parse(
model="gpt-5.5",
input=[{"role": "user", "content": "Coffee with a client, 38 lei."}],
text_format=Expense,
)
expense = resp.output_parsed # already a validated Expense instance
print(expense.amount, expense.category) # 38.0 Category.food
You define the schema once as a Pydantic model, and you get back a typed, validated object — no parsing, no json.loads, no field-existence checks. This is the approach to default to when your provider supports it.
One caveat you must keep in mind: schema conformance is not the same as correctness. Constrained decoding guarantees the output fits the shape — it cannot guarantee the model extracted the right value. It will never again hand you a broken brace, but it can still put 48 where the receipt said 38. Structure is a data-integrity guarantee, not a factual one — which is precisely why you still need evaluation and validation around extraction tasks, the domain of a course on LLM evals and pre-deployment quality.
Approach 4 — Grammar-constrained decoding for local and open models
Running open models (Llama, Mistral, Qwen and friends) via llama.cpp, vLLM, or Ollama? You get the same guarantee with grammar-constrained decoding, and often more control. Libraries like Outlines and llama.cpp's GBNF grammars let you constrain generation to a JSON Schema, a regular expression, or even an arbitrary context-free grammar.
# Outlines — constrain a local model's output to a Pydantic schema.
import outlines
from pydantic import BaseModel
class Expense(BaseModel):
amount: float
currency: str
model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.3")
generator = outlines.generate.json(model, Expense)
expense = generator("Coffee with a client, 38 lei.")
print(expense.amount) # 38.0
Same principle as Approach 3 — the decoder is masked to valid tokens — but running fully under your control, offline, with no per-token API cost. This matters when data can't leave your infrastructure. Wiring constrained decoding into a self-hosted stack is exactly the kind of production integration covered in a course on advanced LLM integration for real applications.
The hard cases: nested schemas, enums, and optional fields
Simple flat objects are the easy demo. Real extraction schemas are nested, have enums, optional fields, and lists — and this is where the difference between the approaches gets sharp.
Consider extracting a full invoice: a header, a customer object, and a variable-length list of line items, each with its own fields. With free-form prompting, the failure rate on a schema this deep is high enough to be unusable — the model loses track of nesting, forgets a required key three levels down, or returns nine line items when the document had ten. With schema-constrained decoding (Approach 3 or 4), the depth stops mattering: the grammar enforces the entire nested structure token by token.
from pydantic import BaseModel
class LineItem(BaseModel):
description: str
quantity: int
unit_price: float
class Customer(BaseModel):
name: str
vat_id: str | None = None # optional — model may legitimately omit it
class Invoice(BaseModel):
invoice_number: str
customer: Customer
items: list[LineItem] # variable-length, each item schema-enforced
total: float
A few nuances that bite even with a schema guarantee:
- Optional vs. required is a modeling decision, not a model decision. If
vat_id is genuinely sometimes absent, mark it optional (str | None). If you mark everything required, a constrained decoder will invent a value to satisfy the schema rather than admit it doesn't know — you've forced a hallucination by over-specifying.
- Enums are your friend. Constraining
category to an enum of four values means the model cannot return a fifth. This is one of the highest-leverage things you can do: every free-text field you can turn into an enum is a class of garbage you've eliminated.
- Provider schema subsets differ. Not every JSON Schema feature is supported by every provider's strict mode (some restrict regex patterns, certain
anyOf compositions, or recursive schemas). Check the provider's documented subset before assuming an exotic schema will be honored.
Getting these modeling decisions right — where to allow null, what to make an enum, how deep to nest — is the difference between an extraction pipeline that degrades gracefully and one that fabricates data on edge cases. It's unglamorous schema design, and it's exactly the kind of production detail a course on building AI applications in Python with real SDKs treats as a first-class skill.
A decision guide
Cut through the options with this:
| Your situation | Use |
|---|
| Hosted model, need guaranteed schema, typed objects | Structured Outputs (Approach 3) — the default |
| Model needs to choose whether/which action to take | Tool calling (Approach 2) |
| Just need "definitely parseable JSON," shape is loose | JSON mode (Approach 1) |
| Self-hosted / open model, data can't leave your infra | Grammar-constrained decoding (Approach 4) |
And regardless of which you pick, keep two habits:
- Validate anyway. Even with a schema guarantee, run the output through Pydantic (or your language's equivalent). It's your last line of defense and it costs nothing. The Instructor library wraps exactly this pattern — schema + validation + automatic retry on failure — across many providers, and is worth adopting.
- Separate "valid" from "correct." A schema guarantee removes an entire class of bugs (parse and shape errors). It does nothing for extraction accuracy. Treat those as two different problems, because they are.
The mindset shift
The reason the broken-JSON problem feels endless is that most developers keep trying to solve it at the parsing layer — sanitizing, regexing, retrying on strings the model already emitted. Every approach above solves it at the generation layer instead: constrain what the model is allowed to produce, and invalid output stops existing rather than getting cleaned up.
That reframing — "constrain generation, don't sanitize output" — is one of those small ideas that quietly removes a whole category of production incidents. It also happens to be the same principle behind reliable prompting, reliable agents, and reliable tool use in general: decide the contract up front and make the model honor it, rather than negotiating with prose after the fact. If you want that foundation built deliberately, it starts with a course on prompt engineering and runs straight through to production integration.
Conclusion
Broken JSON from an LLM is not a mysterious failure you have to defend against forever — it's the predictable result of asking a free-form generator for a structured artifact without constraining it. Pick the right layer: JSON mode for loose parseability, tool calling when the model must choose to act, Structured Outputs for guaranteed typed schemas, grammar-constrained decoding when you run the model yourself. Then validate the result and keep "valid" and "correct" as separate concerns.
Do that, and try/except json.loads() with a regex stops being your JSON strategy — because you no longer have a JSON problem to strip your way out of.
Sources & further reading:
This article is educational content. Provider APIs and library interfaces evolve; verify current syntax and model names in the official documentation before building production systems.
