Tool Calling With Local Models: Making Agents Stop Breaking on Malformed JSON

Written by Jakub Rusinowski · Last updated 2026-08-04 · Hardware figures computed by our VRAM engine

Tool calling is where local coding agents break most often — not because the model cannot reason, but because it emits JSON the harness cannot parse. Three fixes, in order of impact: use a model whose chat template has native tool-call support, constrain decoding with a JSON-schema grammar so invalid output is impossible by construction, and keep the tool surface small. Together these turn a flaky agent into a dependable one without changing the model tier.

An agent is a model that can act. The mechanism is unglamorous: the model emits a structured call — a name and a JSON argument object — and the harness runs it. Every capability you care about, from reading a file to running your test suite, passes through that one narrow interface.

Frontier models make it look trivial. Local models expose how narrow it really is. A 8–32B model asked to produce a tool call will, at some rate, wrap the JSON in an explanation, use file where the schema said path, emit two calls where one was allowed, or start a call and drift into prose. The harness sees an unparseable action, and your agent stalls, retries, or dies. This is not a reasoning failure and a bigger model is not the only fix.

Three ways a local model can call a tool

Knowing which mechanism your stack uses explains most of its failures.

1 · Native tool calling via the chat template. The model was fine-tuned to emit calls in a specific format, and the GGUF's chat template encodes it. The server parses that format and returns a structured tool_calls field in the OpenAI-compatible response. This is the reliable path — and it is why models advertised as *agentic* or *coder* variants behave so much better in agents than general chat models of the same size.

2 · Prompted JSON. The harness describes the tools in the system prompt and asks for JSON back, parsing it itself. Works with any model; fails at a much higher rate, because nothing enforces the shape. Most "why did my agent stop?" reports live here.

3 · Edit formats instead of tool calls. Aider's approach: rather than a tool schema, ask for a diff or a search/replace block. It is a *text* format, so weaker models handle it better than nested JSON, and it fails visibly (the patch does not apply) rather than silently. This is a genuine reason Aider punches above its weight on small models.

Prefer 1, fall back to 3, treat 2 as the thing to constrain.

Grammar-constrained decoding: the fix most people skip

The strongest available fix does not ask the model to behave — it makes misbehaviour impossible. At each token, constrained decoding masks out every token that could not continue a valid output under a formal grammar. Invalid JSON is not corrected after the fact; it is never generated.

llama.cpp uses GBNF (its BNF-style grammar format) and converts a JSON Schema to a grammar automatically, so in practice you hand it the same schema you would give the tool definition. The OpenAI-compatible servers built on it expose this as structured output / JSON-schema response formats; vLLM and SGLang have equivalents.

# llama.cpp server: force every response to match a tool-call schema
llama-server -m qwen3-coder-8b-q4_k_m.gguf --port 8080 --ctx-size 16384

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role":"user","content":"Read src/auth.py"}],
  "response_format": {
    "type": "json_schema",
    "json_schema": {"name":"tool_call","schema":{
      "type":"object",
      "properties":{
        "tool":{"type":"string","enum":["read_file","edit_file","run_tests"]},
        "args":{"type":"object"}
      },
      "required":["tool","args"],
      "additionalProperties": false
    }}
  }
}'

Two caveats worth knowing before you turn it on everywhere:

• There is a small speed cost. Masking adds per-token overhead — typically low single-digit percent in llama.cpp-based runtimes. That is trivially repaid by not re-running failed parses.
• Over-constraining can hurt quality. Forcing a model into a rigid structure at every token can suppress the reasoning it would otherwise do first. The practical shape: let the model think in free text, then constrain only the final action block, or give it an explicit reasoning field inside the schema before the tool field.

Pick a model that was trained for this

Tool-call reliability is a training property, not an emergent one. Two models of identical size can differ enormously, and the difference tracks whether the model was post-trained on agentic traces and diverse tool-call formats.

What to look for, in order:

1. An explicit agentic/coder tune. Devstral-2 22B is trained for agentic software engineering; the Qwen3-Coder line is trained for coding-agent use and tool calling. Both behave far better in a harness than a general chat model of the same parameter count. 2. A chat template with tool support. Check the GGUF actually ships one — ollama show --modelfile <tag> reveals the template. No tool tokens in the template means you are on the prompted-JSON path whether you meant to be or not. 3. Recency over size, within a tier. Tool-calling quality has improved much faster than raw reasoning across 2025–2026. A current 22B usually beats a two-year-old 34B inside an agent loop.

The per-tier picks and exact VRAM figures are in best local coding models. The short version for agents: 8B works with constraints and a small tool set; 22B is the first comfortable tier; 27–32B is where you stop thinking about it.

Models that behave in an agent loop

The tiers where tool calling is dependable enough to leave running. VRAM figures are Q4_K_M weights from the site’s compute engine — add KV cache for your context on top.

Keep the tool surface small

Every tool you expose is described in every request, and every extra option is another chance for a weak model to choose wrong. Both costs are worse locally: tokens are scarce because the window is small, and selection accuracy is lower to begin with.

• Six to eight tools is plenty for coding work: read, edit, list/search, run command, run tests, and maybe fetch. If your registry is at twenty, cut it.
• Connected MCP servers are tools. Several servers can inject tens of thousands of tokens of schemas before you type a word — a third of a large window, and more than all of a small one. Connect per project, not globally (MCP with local agents).
• Give tools disjoint jobs. search_files and grep_repo doing nearly the same thing costs tokens and creates coin-flips. Merge them.
• Descriptions are prompts. One line, imperative, stating exactly when to use it and what it returns. Say what it does *not* do if the model keeps misusing it.

Repair strategies for when it still breaks

Even with constraints, occasional calls fail. Handle them in the harness rather than escalating to a human:

1. Parse-repair retry. On a parse failure, re-prompt with the raw output and the schema, asking only for corrected JSON. This usually succeeds first try and should *not* count against the loop's iteration cap — it is a formatting error, not a failed attempt. 2. Validate arguments, don't just parse. Valid JSON with a hallucinated path is worse than a parse error, because it executes. Check the path exists and is inside the allowed root, that enums are members, and that required fields are present — then return the error message to the model as a tool result. Models correct on specific feedback remarkably well. 3. Lower temperature for action turns. Structured output wants near-greedy decoding. 0.0–0.2 for tool-calling turns; save the higher temperature for prose. 4. Log every malformed call. A rate above roughly one in twenty is a configuration problem — wrong template, unconstrained decoding, too many tools — not bad luck. The log tells you which. 5. Cap the repair loop. Two failed repairs means stop and surface it. Three models arguing with a schema is a wasted evening.

Diagnosing the five common breakages

The pattern worth internalising: formatting failures are configuration bugs, and configuration bugs are cheap to fix. Only after the tool-call rate is clean does a model-tier upgrade tell you anything real.

Frequently asked questions

Keep going

• Reliable JSON & structured output from local models
• Local OpenAI-compatible API servers
• Glossary — constrained decoding, GBNF, function calling
• VRAM checker — sizing the model you picked
• The Agent Harness: What Actually Turns a Local Model Into a Coding Agent
• Loop Engineering on a Local LLM: Agent Loops That Check Their Own Work
• Best Local Coding LLMs in 2026, by VRAM Tier
• MCP With Local Coding Agents: Which Servers Earn Their Tokens
• Local AI coding assistant — the full hub