Context Engineering for Local Coding Agents: Budgeting a Window You Pay For

Written by Jakub Rusinowski · Last updated 2026-08-04 · Hardware figures computed by our VRAM engine

Context engineering means curating the smallest high-signal set of tokens the agent sees each turn. Locally it has a second dimension API users never face: context is VRAM. Every token of window costs KV cache that competes with the weights, so the job is to budget a window you are paying for in hardware — and to compact before quality degrades, which on a local model happens well before the window is full.

On a hosted model, context is an abstract limit with a price per token. On your own GPU it is physical: the KV cache for a long agent conversation is gigabytes of the same VRAM holding the weights. Doubling the context can mean dropping a quantisation level or a model tier — a trade no API user ever has to make.

That changes the discipline. Cloud advice says "you have 200K, use it." Local advice is the opposite: decide your window from your VRAM first, then fit the agent's working set into it. Get this backwards and you meet the two most common local-agent failures — silent prompt truncation, and out-of-memory errors halfway through a task that started fine.

What is actually in the window

Before optimising, know the bill. A typical VS Code agent turn, mid-task, on a repo of moderate size:

Two lines dominate: tool schemas and accumulated tool results. Neither is your prompt, and neither is visible in the chat UI — which is exactly why people conclude the model is dumb when their window is 70% overhead.

The VRAM cost of context

KV cache scales linearly with context length and with model size, and it is additive on top of the weights. The rough shape on a 24 GB card running a 27B at Q4: the weights take roughly three-quarters of the card, and each additional 8K of context takes a further slice — which is why 32K fits and 128K does not, on a model whose card advertises 256K.

Rather than trusting a table, model it: the VRAM calculator computes weights plus KV cache for a specific model, quantisation and context length, using the same engine as every figure on this site. Three practical consequences:

• The advertised context is not your context. A 256K-native model on a 16 GB card is a 16K–32K model in practice. Nothing is wrong; you simply cannot afford the cache.
• KV-cache quantisation buys real room. llama.cpp and its downstreams can store the cache at 8-bit, roughly halving its footprint for a small quality cost — often the difference between 16K and 32K on the same card.
• Longer context slows every turn. Prefill grows with the prompt, so a bloated window costs latency on every single iteration of a loop, not just memory.

If the numbers say your target context does not fit, the honest options are a smaller quant, a smaller model, or renting a bigger GPU — not hoping the truncation goes unnoticed.

Context rot: why quality drops before the window is full

Filling a window does not degrade a model gracefully at 100% and perfectly below it. Attention quality falls as the window fills and as irrelevant content accumulates — the pattern usually called context rot. Local models, generally smaller and often trained with less long-context data than frontier models, hit it earlier: a practical rule is that a local coding agent starts getting noticeably worse somewhere around half to two-thirds of its served window.

Symptoms, in the order they usually appear:

1. It stops honouring the rules file (those tokens are furthest back and most diluted). 2. It re-reads files it already read, because the earlier read has been buried. 3. It references functions that do not exist — often ones from an earlier, discarded plan. 4. Edits start conflicting with edits it made twenty turns ago.

The fix is not a bigger window. It is removing tokens: compaction, subagents, or a restart. A fresh session with a one-paragraph handoff routinely outperforms a degraded session with full history — and locally it costs nothing but the prefill.

Five techniques, ordered by payoff

1 · Set the served window deliberately. Not a context technique so much as the precondition for all of them. Ollama truncates to its default silently; a long-context variant is two lines:

cat > Modelfile <<'EOF'
FROM qwen3.6:27b
PARAMETER num_ctx 32768
EOF
ollama create qwen3.6:27b-32k -f Modelfile

Then select that variant in the agent, and confirm the cache actually fits with the checker before trusting it.

2 · Truncate tool results at the source. A full test log or a recursive listing can be tens of thousands of tokens of noise. Keep the tail of failures (that is where the assertion is), cap greps to the matching lines with a little surrounding context, and never let a raw find or ls -R into the window. Most harnesses expose a limit; set it to something your window can survive.

3 · Compact deliberately, not at 95%. Production harnesses compact in stages — drop stale tool output, then summarise old turns, then rebuild the working set. Whatever your tool offers, trigger it earlier than the default: thresholds designed for a 200K frontier window are far too late for a local 32K one. What must survive a compaction: the goal, decisions already made, the current diff, and open problems. What should not: raw logs, superseded plans, file contents already edited.

4 · Curate the working set by hand. Adding files explicitly beats letting the agent discover them — it is faster, cheaper, and dramatically improves small-model accuracy. Aider's /add and /drop, Cline's explicit file mentions, and per-project ignore rules are the primitives. Three right files beat thirty plausible ones.

5 · Isolate with subagents or separate sessions. The pattern that scales: a sub-task runs in its own window and returns a short summary rather than its whole transcript. Even without built-in subagent support you can do this manually — run the noisy exploration ("which files touch auth?") in one session, then start the edit session with the three-line answer. This is the single most effective local technique for long tasks, because it keeps the main window small permanently.

Retrieval vs the whole repo

Long-context marketing suggests pasting the codebase in. For local agents this is the wrong instinct twice over: you cannot afford the cache, and accuracy usually *drops* when the relevant file is buried among fifty irrelevant ones.

The working alternative is search, not stuffing. Coding agents that lean on grep/ripgrep and the file tree — reading only what the search returns — tend to outperform ones that pre-load context, and they are far cheaper per turn. Practically:

• Let the agent search, then read the two or three files it found.
• Keep a repo map (a compact list of files with their key symbols) instead of file contents.
• If you want semantic search over a large codebase, index it separately and feed back snippets — the local RAG pipeline applies unchanged to code.

Whole-file context earns its place for the file being edited, and rarely beyond it.

A budget that works on 16 and 24 GB

A concrete starting allocation, assuming a 16K served window on 16 GB and 32K on 24 GB:

If the schemas alone break the first row — the common case with several MCP servers connected — that is your fix, and it is a bigger win than any prompt rewrite. Details in MCP with local coding agents.

Frequently asked questions

Keep going

• Context windows explained — the fundamentals
• Context length & KV-cache OOM — diagnosis and fixes
• VRAM checker — weights plus KV cache for your exact setup
• Local RAG pipeline — retrieval instead of stuffing
• The Agent Harness: What Actually Turns a Local Model Into a Coding Agent
• Loop Engineering on a Local LLM: Agent Loops That Check Their Own Work
• MCP With Local Coding Agents: Which Servers Earn Their Tokens
• AGENTS.md for Local Coding Agents: The Cheapest Quality Upgrade You Can Make
• Local AI coding assistant — the full hub