The Agent Harness: What Actually Turns a Local Model Into a Coding Agent

Written by Jakub Rusinowski · Last updated 2026-08-04 · Hardware figures computed by our VRAM engine

A harness is the software around the model that makes it an agent: it assembles the prompt, dispatches tool calls, feeds results back, compacts context, enforces permissions, and persists state between turns. The model supplies judgement; the harness supplies everything else. On local hardware the harness is the higher-leverage half — the same 27B weights behave like a different model under a good one, and harness improvements cost VRAM nothing.

Ask what makes Claude Code, Cline or Aider good and the answer people expect is "the model." Run the same weights through a bare chat box and you find out otherwise. Benchmarks tell the same story from the other direction: identical open weights score dramatically differently depending on the agent scaffold they are evaluated under, which is why a model card's SWE-bench number always names the harness it was measured with.

For local setups this is the most useful fact in the field. You cannot download a better model into a 16 GB card, but you *can* give the model you already have better tools, tighter context and cleaner feedback — and that is a bigger delta than one quantisation step in either direction. Harness engineering is the part of agent quality that is not bounded by your VRAM.

Harness vs scaffold vs model: getting the words right

The three terms get used interchangeably and they are not the same thing. The distinction that has settled in 2026:

In casual use "harness" covers scaffold too, and that's fine — but the split is worth keeping when debugging. *The agent used the wrong file* is usually a scaffold problem (bad context assembly, missing rules). *The agent crashed on a malformed tool call, or forgot the instructions at turn 30* is a harness problem (parsing, compaction). They have different fixes.

The six jobs, and how each one fails on local models

Every harness — hand-rolled or off the shelf — does these six things. Each has a characteristic local-model failure worth recognising by sight.

1 · Context assembly. Builds the prompt: system prompt, rules file, the tools' schemas, the files it thinks are relevant, the conversation so far. *Local failure:* the assembled prompt silently exceeds the served context window and the front of it — including your instructions — is dropped. This produces the classic "it ignored my rules" report, and the fix is a server-side context setting, not a better prompt. Covered in context engineering.

2 · Output parsing. Turns the model's reply into a structured action. *Local failure:* the model wraps JSON in prose, invents an argument name, or emits two calls where the schema allows one. Grammar-constrained decoding removes most of this class outright — see tool calling.

3 · Tool dispatch. Actually runs the action — read file, apply edit, run shell command, query an MCP server — and formats the result back into context. *Local failure:* tool *result* bloat. A ls -R or a full test log can be tens of thousands of tokens; a good harness truncates and summarises before the result ever reaches a model with a 32K window.

4 · Permission enforcement. Decides what may run unattended and what needs a human. *Local failure:* approval fatigue — after the twentieth prompt everyone clicks yes without reading, which is functionally the same as having no gate. Fix it structurally with an allowlist plus a sandbox rather than with more prompts (sandboxing).

5 · Context management. Decides what to keep, compact, or offload as the window fills. Production harnesses do this in stages — trimming stale tool output first, summarising older turns next, and rebuilding the working set only when they must. *Local failure:* nothing compacts, the window fills, quality falls off a cliff mid-task. Local models degrade earlier in the window than frontier ones, so compaction thresholds tuned for a 200K API context are wrong for your 32K.

6 · State persistence. Keeps the session across restarts: what changed, what's approved, what the task is. *Local failure:* usually the harness handles this fine — the risk is the opposite, resuming into a stale, poisoned context that would have been better restarted.

Harness engineering for people who use agents (not build them)

Most readers will never write a harness. You will still do harness engineering, because every knob a coding agent exposes is a harness knob. Ranked by return on effort for local models:

1. Set the served context window explicitly. The single highest-value change, and the one most people miss. Ollama's default context truncates agent prompts silently. Create a variant with num_ctx set to what your VRAM can defend (16K on 16 GB, 32K on 24 GB) and select *that*. Everything else on this list is worthless if the harness's prompt is being cut in half. 2. Write a rules file. AGENTS.md at the repo root — build command, test command, layout, conventions, what never to touch. Small models depend on it far more than large ones, because they cannot infer conventions from a few files. See AGENTS.md for local agents. 3. Cut the tool surface. Every tool definition is tokens in every request, and every extra tool is another chance for a weak model to pick wrong. Ten MCP servers can consume a third of a large context window before you type anything — locally that is fatal. Keep the tools the task needs. (MCP with local agents.) 4. Give it a verifier and let it check itself. Configure the test command the agent runs after edits. This converts a level-1 harness into a level-2 loop — see loop engineering. 5. Scope the working set. Add files deliberately; don't let the agent index a monorepo into context. Aider's /add, Cline's file mentions, and .gitignore-style excludes all exist for this. 6. Start fresh often. A new session is free and beats fighting a degraded one. Treat "restart with a tighter task" as the first debugging step, not the last resort.

Build a minimal harness in about 60 lines

Worth doing once, even if you go back to Cline afterwards — it makes every configuration option in a real harness legible. This one talks to any OpenAI-compatible local server, exposes two tools, and enforces a trivial permission rule.

# harness.py — the smallest thing that is honestly an agent harness
import json, subprocess, pathlib, urllib.request

ENDPOINT, MODEL = "http://localhost:11434/v1/chat/completions", "qwen3-coder:8b"
ROOT = pathlib.Path("src").resolve()          # permission boundary
MAX_TOOL_CHARS = 6000                         # tool-result truncation
MAX_TURNS = 12

TOOLS = [
  {"type": "function", "function": {"name": "read_file",
    "description": "Read a UTF-8 text file inside src/.",
    "parameters": {"type": "object", "required": ["path"],
      "properties": {"path": {"type": "string"}}}}},
  {"type": "function", "function": {"name": "run_tests",
    "description": "Run the project test suite. Returns pass/fail and output.",
    "parameters": {"type": "object", "properties": {}}}},
]

def dispatch(name, args):
    if name == "read_file":                                   # 4 · permission gate
        p = (ROOT / args["path"]).resolve()
        if ROOT not in p.parents and p != ROOT:
            return "DENIED: path outside src/"
        return p.read_text()[:MAX_TOOL_CHARS]                 # 3 · result truncation
    if name == "run_tests":
        r = subprocess.run(["pytest", "-q"], capture_output=True, text=True)
        return f"exit={r.returncode}\n" + (r.stdout + r.stderr)[-MAX_TOOL_CHARS:]
    return f"unknown tool {name}"

def chat(messages):                                            # 1 · context assembly
    body = json.dumps({"model": MODEL, "messages": messages,
                       "tools": TOOLS, "temperature": 0.1}).encode()
    req = urllib.request.Request(ENDPOINT, body, {"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        return json.load(r)["choices"][0]["message"]

msgs = [{"role": "system", "content": "You fix bugs. Use tools. Stop when tests pass."},
        {"role": "user", "content": "The suite is red. Diagnose and report the cause."}]

for turn in range(MAX_TURNS):
    m = chat(msgs)
    msgs.append(m)
    calls = m.get("tool_calls") or []                           # 2 · output parsing
    if not calls:
        print(m.get("content", "")); break
    for c in calls:                                            # 3 · tool dispatch
        out = dispatch(c["function"]["name"], json.loads(c["function"]["arguments"] or "{}"))
        msgs.append({"role": "tool", "tool_call_id": c["id"], "content": out})
    if len(json.dumps(msgs)) > 60_000:                          # 5 · crude compaction
        msgs = msgs[:2] + [{"role": "user", "content": "…earlier steps elided…"}] + msgs[-6:]

That is jobs 1–5 in miniature; job 6 is one json.dump away. What a production harness adds is not conceptual — it is a hundred careful decisions about *which* tool, *how much* truncation, *when* to compact and *what* to summarise. Those decisions are where the quality is.

Choosing a harness for a local model

You are picking a harness far more than you are picking a UI. What matters locally: how big is its system prompt, how many tools does it define by default, can you point it at an OpenAI-compatible endpoint, and does it compact context sensibly.

The heuristic: the smaller your model, the lighter the harness should be. A large system prompt plus a dozen tool schemas can consume a quarter of a 16K window before the task is stated — and an 8B model has fewer tokens of attention to spare on it. Aider's design (a compact repo map and diff-based editing rather than a wide tool surface) is not old-fashioned; it is the design that survives on modest hardware. Full comparison in the tool matrix.

How to tell whether the harness or the model is your problem

A short diagnostic, in order. Each step isolates one layer.

1. Does the raw model answer the question at all? Paste the task and the relevant file into a plain chat with the same model. Good answer → the model is fine, the harness is starving or confusing it. Bad answer → model tier is the constraint; more harness will not save it. 2. Is the prompt arriving intact? Log the harness's request, or run ollama ps / check your server logs for the active context size. If the assembled prompt exceeds it, everything else is noise until you fix it. 3. Are tool calls parsing? Count the malformed-call retries. More than the occasional one means a decoding-constraint problem, not an intelligence problem. 4. Does quality fall off a cliff at a particular point? That is context management. Compact earlier, or restart sessions more often. 5. Does it do the right thing but in the wrong place? Scaffold: your rules file and working set need work, not the model.

Only after all five should you conclude you need a bigger model — and by then you will know precisely which tier, because you will know which layer ran out of room.

Why harness work compounds and model upgrades do not

A better GPU improves everything by a fixed amount, once, for a fixed price. A better harness improves everything by a compounding amount, permanently, for the cost of an afternoon — and it carries over when you *do* upgrade, and again when a better open model lands next quarter.

That asymmetry is the argument for treating the harness as the primary artefact of a local coding setup: the rules file, the tool allowlist, the verifier command, the context settings, the sandbox. Those files are yours. The weights are replaceable.

Frequently asked questions

Keep going

• Local OpenAI-compatible API servers — what the harness talks to
• Context length & KV-cache OOM
• Building autonomous agents — the general agent guide
• VRAM checker — sizing model plus context together
• Loop Engineering on a Local LLM: Agent Loops That Check Their Own Work
• Context Engineering for Local Coding Agents: Budgeting a Window You Pay For
• Tool Calling With Local Models: Making Agents Stop Breaking on Malformed JSON
• Continue vs Cline vs Aider vs Tabby: the Local Coding Tool Matrix
• Local AI coding assistant — the full hub