Written by Jakub Rusinowski · Last updated July 10, 2026
To get reliable JSON from a local LLM, stop begging in the prompt and constrain the decoder: Ollama's structured outputs force the model to emit only tokens that fit your schema, making malformed JSON
To get reliable JSON from a local LLM, stop begging in the prompt and constrain the decoder: Ollama's structured outputs force the model to emit only tokens that fit your schema, making malformed JSON mechanically impossible. This guide covers Ollama's format parameter, GBNF grammars in llama.cpp, LM Studio's schema mode, and the validate-and-retry pattern that makes the whole thing production-safe.
Last Updated: July 2026
"Respond ONLY with valid JSON" is a request, not a constraint. The model *samples* every token, so given enough calls it will eventually produce a trailing comma, a chatty preamble ("Sure! Here's your JSON:"), markdown fences around the object, or a cut-off brace at the context limit. At temperature > 0 this isn't a bug — it's what sampling means. Anything you build on top (parsers, agents, pipelines) then dies on JSONDecodeError.
The fix is grammar-constrained decoding: at each step, the runtime masks away every token that would violate your schema, and the model can only choose among legal continuations. Prompt engineering reduces failures; constrained decoding eliminates the malformed-syntax class entirely.
The quickest win — force syntactically valid JSON, any shape:
curl http://localhost:11434/api/chat -d '{
"model": "llama3.1",
"format": "json",
"stream": false,
"messages": [{"role": "user",
"content": "Extract name and email from: Reach Jan Kowalski at jan@example.com. Reply in JSON."}]
}'Two rules: always also say "reply in JSON" in the prompt (otherwise some models emit whitespace forever), and remember this only guarantees *valid* JSON — not *your* JSON. The keys might still be anything. For that, you need a schema.
Since Ollama 0.5, format accepts a full JSON schema — this is the setting you actually want in apps:
curl http://localhost:11434/api/chat -d '{
"model": "llama3.1",
"stream": false,
"messages": [{"role": "user", "content": "Extract: Reach Jan Kowalski at jan@example.com"}],
"format": {
"type": "object",
"properties": {
"name": {"type": "string"},
"email": {"type": "string"}
},
"required": ["name", "email"]
}
}'In Python, let Pydantic write the schema and validate the result — this pair is the core of every reliable local-LLM integration:
from ollama import chat
from pydantic import BaseModel
class Contact(BaseModel):
name: str
email: str
resp = chat(
model="llama3.1",
messages=[{"role": "user", "content": "Extract: Reach Jan Kowalski at jan@example.com"}],
format=Contact.model_json_schema(),
options={"temperature": 0},
)
contact = Contact.model_validate_json(resp.message.content)Note temperature: 0 — for extraction and classification you want determinism, and it measurably improves field-level accuracy on 7–8B models. Docs: Ollama structured outputs.
Ollama's schema support is implemented on top of llama.cpp's GBNF grammars — a BNF dialect that can constrain output to *any* formal shape, not just JSON (SQL fragments, ini files, one-of-N classifier labels):
# choice.gbnf - the model can output ONLY one of three labels
root ::= "positive" | "negative" | "neutral"./llama-cli -m model.gguf --grammar-file choice.gbnf \
-p "Classify the sentiment: 'The delivery was late again.'"For JSON specifically, llama.cpp ships json.gbnf and a json_schema_to_grammar.py converter. Use raw GBNF when you're on llama.cpp/LM Studio's server anyway, or when the output isn't JSON at all. Reference: llama.cpp grammars.
LM Studio's local server accepts the OpenAI response_format parameter, so existing OpenAI SDK code ports directly:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")
resp = client.chat.completions.create(
model="qwen2.5-7b-instruct",
messages=[{"role": "user", "content": "Extract: Jan Kowalski, jan@example.com"}],
response_format={"type": "json_schema", "json_schema": {"name": "contact", "schema": {
"type": "object",
"properties": {"name": {"type": "string"}, "email": {"type": "string"}},
"required": ["name", "email"]}}},
)The same shape works against vLLM if you graduate to production serving.
Tool/function calling is structured output wearing a trench coat: the model emits JSON naming a function and its arguments. Ollama supports a tools parameter on capable models (Llama 3.1+, Qwen 2.5+, Mistral). The reliability rules are identical — small models pick the *right* tool less reliably than they format the call, so keep tool counts low (3–5) and descriptions sharp. If you're building agents on top, see Building Autonomous Agents.
Constrained decoding kills syntax errors; it can't stop the model from putting the wrong *content* in the right shape (empty strings, hallucinated emails). Production code wraps every call in validation:
def extract(text, retries=2):
for attempt in range(retries + 1):
resp = chat(model="llama3.1", format=Contact.model_json_schema(),
options={"temperature": 0},
messages=[{"role": "user", "content": f"Extract contact: {text}"}])
try:
c = Contact.model_validate_json(resp.message.content)
if "@" in c.email: # semantic check, not just shape
return c
except ValidationError:
pass # schema drift - retry
return None # explicit failure beats silent garbageThree habits that keep failure rates near zero: validate *semantics* (not just parseability), fail explicitly to a None/queue instead of guessing, and log every retry — a rising retry rate is how you notice a model or prompt regression.
| Model | Structured output quality |
|---|---|
| Qwen 2.5 7B/14B | Best-in-class at following schemas; the default choice |
| Llama 3.1 8B | Reliable with constrained decoding; strong tool calling |
| Mistral NeMo 12B | Good, occasionally over-nests without constraints |
| 3B-class models | Fine *with* constrained decoding; hopeless without |
With grammar constraints active, even small models produce perfect syntax — the differences move to content accuracy, where Qwen 2.5 leads at extraction tasks.
Because you're relying on the prompt alone. Sampling means occasional format drift is guaranteed at scale. Switch to constrained decoding — format in Ollama, response_format in LM Studio, or a GBNF grammar in llama.cpp — and syntactically invalid output becomes impossible.
Heavy constraints can slightly reduce answer quality on reasoning-type tasks — the model can't "think out loud" before committing to structure. For extraction and classification the effect is negligible. If you need reasoning plus structure, do two calls: reason freely first, then extract into the schema.
Marginally — the grammar mask adds per-token overhead, typically single-digit percent in llama.cpp-based runtimes. You'll win it back many times over by never re-running failed parses.
Yes, tokens stream as usual — but the object is only parseable when complete. Practical patterns: validate at end-of-stream, or stream to the UI while your code waits for the final brace.