Why Your Local Coding Agent Feels Slow — and the Six Fixes That Work

Written by Jakub Rusinowski · Last updated 2026-08-04 · Hardware figures computed by our VRAM engine

A local coding agent usually feels slow because of prefill, not generation: every turn re-processes a large prompt before the first new token appears. The fixes in order of payoff are prompt/KV cache reuse so unchanged prefixes are not recomputed, a smaller working context, speculative decoding for 1.5–3× faster generation, keeping the model resident, and running a separate small model for autocomplete. Together they change the experience more than one GPU tier does.

The complaint is almost never "tokens per second are low." It is "I ask for something and nothing happens for twenty seconds." Those are different problems with different fixes, and confusing them sends people shopping for a GPU they may not need.

Every agent turn has two phases. Prefill processes the whole prompt — system prompt, tool schemas, rules file, every file in context, the entire conversation so far — before a single output token exists. Decode then produces the answer one token at a time. Chat is decode-dominated, so tokens/sec is the number that matters there. Agent work is prefill-dominated, because the prompt is enormous and the reply is often a short tool call. That inversion is why an agent can feel sluggish on hardware that chats comfortably, and why the biggest wins come from not recomputing what you already computed.

Fix 1 — Reuse the prompt cache (largest single win)

An agent's prompt is mostly identical from turn to turn: same system prompt, same tools, same rules file, same files, plus a little new conversation at the end. Recomputing that unchanged prefix every turn is pure waste, and it is exactly what happens by default in some setups.

llama.cpp and its downstreams keep the KV cache for a shared prefix and only process the new suffix — turning a multi-second prefill into a fraction of it. What breaks the reuse is the part worth knowing:

Anything that changes early in the prompt invalidates everything after it. A timestamp in the system prompt, a rules file that regenerates, a tool list in nondeterministic order — each one throws away the whole cache every turn.
Compaction resets it too. Necessary, but it means the turn right after a compaction is expensive. Compacting less often but more thoroughly beats trimming constantly.
Alternating between two models on one server evicts the other's cache. Keep one model resident per session.

Put the volatile content — the current task, the latest tool output — at the *end* of the prompt, and keep the front byte-identical. This is free, and on a long session it is the difference between usable and not.

Fix 2 — Stop paying for context you are not using

Prefill time scales with prompt length, so every token you never needed costs you on every single turn. The savings compound across an agent loop in a way they never do in chat.

The three biggest sources, all covered in depth elsewhere in this hub:

Tool schemas from connected MCP servers — potentially tens of thousands of tokens per request, most of them describing tools this task will not call (MCP).
Unbounded tool output — a full test log or recursive listing pasted into context, then re-processed every subsequent turn.
Whole files where a snippet would do (context engineering).

There is also a subtler setting: allocating a 32K context when you use 8K still costs VRAM that could have gone to a better quantisation. Size the served window to what you actually use, and check the trade with the VRAM calculator.

Fix 3 — Speculative decoding

The best generation-side speedup available on local hardware. A small draft model proposes several tokens ahead; the full model verifies them in a single pass, accepting the ones it agrees with. Output is identical to running the big model alone — this is not an approximation — and typical speedups are 1.5–3×.

It works unusually well on code. Code is highly predictable in the small: brackets, indentation, boilerplate, repeated identifiers. A tiny draft model gets long runs of those right, so acceptance rates are high.

# llama.cpp: 27B target with a small same-family draft model
llama-server -m qwen3.6-27b-q4_k_m.gguf \
  -md qwen3-coder-1.5b-q4_k_m.gguf \
  --draft-max 16 --draft-min 4 \
  --ctx-size 32768 --port 8080

Two conditions to check before expecting the win. The draft model must share a tokenizer with the target — in practice, same family, much smaller. And it must fit alongside everything else: a 1.5B at Q4 is a small addition, but on a card that is already full it can force a worse quantisation of the main model, which is a net loss. On tight VRAM, skip it.

Fix 4 — Keep the model resident

A cold start reloads gigabytes of weights from disk. If your agent goes quiet for a few minutes and the server has unloaded, the next request pays for that — and it reads as "the agent hangs sometimes."

# Ollama: keep weights loaded for an 8-hour working session
export OLLAMA_KEEP_ALIVE=8h
# Or per request
curl http://localhost:11434/api/generate -d '{"model":"qwen3.6:27b-32k","keep_alive":"8h"}'

Related: do not swap models mid-session unless you must. Model switching costs a reload plus the loss of the prompt cache — a habit that is free on a hosted provider and expensive on your own GPU. Pick the model for the session, not for the message.

Fix 5 — Two models, two jobs

Autocomplete and agent work have opposite requirements. Completion must answer in tens of milliseconds and only needs local context. Agent work tolerates seconds and needs reasoning. Serving both from one 27B makes completion unusable and wastes the big model on trivial suggestions.

Run a small fill-in-the-middle model — StarCoder2 3B or Qwen3-Coder 8B — for inline completion, and the large model for chat and agent turns. On a 24 GB card both fit comfortably; the small one costs a couple of gigabytes and transforms the moment-to-moment feel of the editor. Setup details are in local Copilot alternatives.

Fix 6 — Match the runtime to the job

Ollama is the easiest way to start and stays fine for single-user work. Two situations justify moving:

A shared team box. vLLM or SGLang batch concurrent requests far better, so four developers hitting one server do not queue behind each other. See the vLLM guide.
You want the tuning knobs. llama-server exposes draft models, cache quantisation, batch sizes and chunked prefill directly. If you are optimising rather than using, go one layer down.

Two settings worth trying wherever your runtime supports them: flash attention, which reduces attention memory and speeds long-context prefill; and 8-bit KV cache, which roughly halves cache footprint, letting you keep a longer window without dropping a quantisation level. Both are usually a flag.

Measure before you buy

Three numbers tell you what to fix, and they take five minutes to collect:

1. Time to first token on a full agent prompt. High and rising with session length → prefill and cache reuse. This is the number that dominates the felt experience. 2. Tokens per second during generation. Low → speculative decoding, a smaller quant, or genuinely more GPU. 3. Time to first token on a *repeat* prompt. If it is the same as a cold one, your prompt cache is not being reused — go back to Fix 1, because everything else is second-order until it is.

Compare against measured throughput for your card and model on the benchmarks page. If your numbers are far below what similar hardware reports, the problem is configuration, not the GPU. If they match and it is still too slow for how you want to work, that is when a hardware conversation is honest — best GPU for AI coding has the tiers, and renting for an evening tests the hypothesis for a few dollars.

No hardware? Rent the GPU first

Before buying a card to fix latency, rent the tier above yours for an evening and run the same session. If time-to-first-token barely moves, your bottleneck was prompt size or cache reuse — and that fix is free.

RunPod — RTX 4090 ≈ $0.3–0.7/hr, A100/H100 by the hour; serverless per-second billing available
Vast.ai — Marketplace of hosted GPUs — often the cheapest per hour (interruptible options)
Lambda — On-demand A100/H100/B200 instances and clusters, per-hour billing

Full list on the cloud AI directory.

Frequently asked questions

Why is my local coding agent so slow to start responding?

Prefill. Every turn re-processes the whole prompt — system prompt, tool schemas, rules file, files in context and the conversation so far — before the first output token exists. Agent prompts are large and replies are often short, so agent work is prefill-dominated in a way chat is not. Reusing the prompt cache and shrinking the prompt attack this directly; a faster GPU only partly does.

What is speculative decoding and does it help for coding?

A small draft model proposes several tokens ahead and the full model verifies them in one pass, giving identical output at typically 1.5–3× the speed. It works especially well on code, which is highly predictable in the small — brackets, indentation, boilerplate — so acceptance rates are high. The draft model must share the target’s tokenizer and must fit in VRAM without forcing a worse quantisation of the main model.

How do I stop my model reloading between requests?

Keep it resident: set OLLAMA_KEEP_ALIVE to your working session length, or pass keep_alive per request. Also avoid switching models mid-session — a switch costs both a weight reload and the loss of the prompt cache, which is free on a hosted provider and expensive on your own GPU.

Does prompt caching work with agents?

Yes, and it is the largest single win — agent prompts are nearly identical turn to turn. The catch is that anything changing early in the prompt invalidates the whole cache after it, so a timestamp in the system prompt or a nondeterministically-ordered tool list can silently cost you every turn. Keep the front of the prompt byte-identical and put volatile content at the end.

Should I use one model for both autocomplete and agent work?

No. They have opposite requirements: completion needs an answer in tens of milliseconds from local context, agent turns tolerate seconds and need reasoning. Run a small FIM-tuned model such as StarCoder2 3B or Qwen3-Coder 8B for inline completion alongside the large model for chat and agent work — on a 24 GB card both fit comfortably.

Will a faster GPU fix a slow coding agent?

Sometimes, but measure first. If time-to-first-token on a repeated prompt is as slow as on a cold one, your prompt cache is not being reused and no GPU will fix that. If your tokens/sec matches what similar hardware reports on the benchmarks page and it is still too slow for how you work, that is when a hardware upgrade is the honest answer — and renting the next tier for an evening tests it cheaply.