Evaluating a Local Coding Agent: Benchmarks Lie, Your Repo Doesn’t

Written by Jakub Rusinowski · Last updated 2026-08-04 · Hardware figures computed by our VRAM engine

Public benchmarks tell you which frontier models lead; they do not tell you whether Devstral-2 22B beats Qwen 3.6 27B on your TypeScript monorepo, because scores are Python-heavy and every number assumes a specific agent harness. Build a small eval instead: twenty real tasks from your own git history, each with a deterministic pass/fail check, run three times per configuration. A weekend of setup, and every model and harness decision afterwards is evidence-based.

Two questions come up constantly and only one has a public answer. "Which open model is strongest?" — the benchmark leaderboards handle that, and our dated Report #2 tracks the landscape. "Will this model do *my* work?" — nothing published answers that, and the gap between the two is where most disappointment with local coding agents comes from.

The gap has three causes. Benchmarks skew heavily Python and heavily toward a particular kind of bug-fix task. Every score is produced by a specific harness, and the same weights score very differently under a different one — which is precisely why the harness is worth engineering. And benchmark runs assume a context length and a quantisation that may be nothing like what your GPU serves.

What the public benchmarks actually measure

Worth knowing so you can read them correctly rather than dismiss them.

Two reading rules. Always check which harness produced the number — a model card's score is a model-plus-scaffold score, and swapping the scaffold moves it. Discount for quantisation and context: published numbers are typically full precision with generous windows, while you are running Q4 at 16–32K. That difference is usually small for straightforward work and grows on long, multi-file tasks.

The reasonable use of leaderboards is to shortlist two or three candidates for your tier. The decision comes from your own eval.

Build a twenty-task eval from your own history

Your git history is a labelled dataset nobody else has: every bug-fix commit is a task with a known-good solution and, if you have tests, a built-in verifier.

1 · Mine the tasks. Pick 20 commits that fixed something and touched 1–3 files. Prefer ones where a test changed alongside the fix — that test is your checker.

git log --oneline --no-merges --grep='^fix' -n 60 --format='%h %s'
git show --stat <sha>          # keep the ones touching 1-3 files

2 · Turn each into a task. For each commit, record: the parent SHA (the starting state), a one-paragraph task description written the way you would brief a colleague — *without* naming the fix — and the check command that must pass afterwards.

tasks/
  001-null-user-session/
    base.sha          # parent commit
    prompt.md         # "Sessions for anonymous users crash on refresh. Fix it."
    verify.sh         # pytest tests/test_session.py -q

3 · Include the boring middle. Twenty tasks that are all hard tells you nothing but "no". Aim for roughly 8 easy (single file, obvious), 8 medium (two or three files, needs a search), 4 hard (cross-cutting). The distribution should look like your actual week.

4 · Add trap tasks. Two or three where the right answer is "this is under-specified" or the fix requires touching a file the rules forbid. Models that confidently do the wrong thing are worse than models that stop, and no public benchmark measures this.

Running it fairly

#!/usr/bin/env bash
# eval.sh — run every task against one configuration, 3 seeds
CONFIG="$1"                                  # label, e.g. devstral22b-aider-16k
for task in tasks/*/; do
  for seed in 1 2 3; do
    git worktree add -f /tmp/ev -b ev-tmp "$(cat "$task/base.sha")" >/dev/null
    ( cd /tmp/ev && timeout 900 run-agent.sh "$task/prompt.md" )
    ( cd /tmp/ev && bash "../../$task/verify.sh" ) && r=PASS || r=FAIL
    echo "$CONFIG,$(basename "$task"),$seed,$r" >> results.csv
    git worktree remove -f /tmp/ev; git branch -D ev-tmp
  done
done

The rules that make results comparable:

• Change one variable at a time. Model, or harness, or context length — never two. Otherwise you learn nothing about which mattered.
• Three runs minimum. Agent runs are high-variance; a single run difference of one or two tasks is noise. If two configurations are within one task of each other, they are tied.
• Fresh worktree per attempt. No state leaks between runs.
• Wall-clock timeout, always. A local agent that takes 40 minutes on a 5-minute task has failed even if it eventually passes.
• Record cost as time and watts, not dollars. Locally the currency is your evening and your electricity — the electricity guide puts numbers on the second one.

Measure more than pass rate

Pass rate alone hides the differences that decide whether you keep using a setup.

• Pass@1 by difficulty bucket. A model that clears every easy task and no hard one is still a useful model — for easy tasks. That is actionable; an aggregate number is not.
• Median turns to green. The best proxy for how it *feels* to use. Two models with the same pass rate and a 3-vs-9 turn difference are not comparable tools.
• Malformed tool-call rate. Above roughly one in twenty, fix the configuration before believing anything else the eval says (tool calling).
• Diff size versus the human fix. Local models often pass by changing far too much. A 200-line diff for a 6-line fix is a fail in review even when the suite is green.
• Cheat rate. How often it edited tests, added any, or skipped assertions. Run git diff --stat -- tests/ on every result; this is the number that separates a useful agent from an impressive one.
• Wall-clock per task. On local hardware this decides what you will actually use.

What an eval typically reveals

Findings that recur often enough to expect:

1. Harness changes beat model changes, up to a point. Raising the served context, adding a rules file and giving the agent a test command routinely move pass rate more than one model tier does — and cost nothing in VRAM. Run those experiments first. 2. The tier below your assumption is often enough. Many teams discover their real distribution is 80% easy-and-medium, where the 16 GB model matches the 24 GB one at twice the speed. 3. Quantisation matters less than context. Q4 versus Q5 is usually within noise on this kind of eval; 8K versus 32K is not. 4. The hard bucket is where local plateaus. Cross-cutting tasks are where frontier models still separate — which tells you exactly where a hybrid fallback earns its keep, and how often you would actually use it. 5. Your eval ages. Re-run it when a new model lands or when you change harness. It takes an hour once it exists, and it turns "the new model feels better" into a number.

A cheaper version for one afternoon

If twenty tasks is too much to start, five is not nothing. Take your last five bug-fix commits, write the prompts, use the existing tests as checkers, run each three times against your current setup, and record pass rate and median turns. That single number is already more informative about your repo than any leaderboard — and it gives you a baseline, which is the part that matters. Every configuration change after that is a comparison instead of a guess.

No hardware? Rent the GPU first

Running the same eval against a model your card cannot hold is the cleanest way to find out whether an upgrade is worth it. Rent the bigger GPU for an evening, run the twenty tasks, and let the pass rate decide.

• RunPod — RTX 4090 ≈ $0.3–0.7/hr, A100/H100 by the hour; serverless per-second billing available
• Vast.ai — Marketplace of hosted GPUs — often the cheapest per hour (interruptible options)
• Lambda — On-demand A100/H100/B200 instances and clusters, per-hour billing

Full list on the cloud AI directory.

Frequently asked questions

Keep going

• Report #2 — the open coding-model landscape (dated, citable)
• Community benchmarks — measured throughput by GPU and model
• Electricity cost of running LLMs — the other half of local cost
• Model library — full specs per family
• Loop Engineering on a Local LLM: Agent Loops That Check Their Own Work
• Best Local Coding LLMs in 2026, by VRAM Tier
• The Agent Harness: What Actually Turns a Local Model Into a Coding Agent
• Tool Calling With Local Models: Making Agents Stop Breaking on Malformed JSON
• Local AI coding assistant — the full hub