How to evaluate a decision model on your own data

Public scores tell you which models are worth testing. Only your own data tells you which one to deploy. Use the community Decision Index and DecisionBench to pick two or three candidates. Then label 200 real examples from your own tickets or emails, run every candidate on the same rows, and compare them pair by pair, on accuracy, calibration, coverage at your threshold, failures and latency. One headline accuracy number hides almost everything that decides whether a model is safe to automate.

Three kinds of evidence, and how far to trust each

Every score on a decision model's page carries a label saying who measured it, because that is the first thing to know about any number:

None of these is wrong, and none answers your question, which is how the model does on your distribution. The hub's other benchmarks table lists every author and harness score in the library with its provenance, and the same model often looks very different from one row to the next.

Benchmark built by the same team that built this model · Run by the authors on a public third-party harness · Authors' own internal benchmark — not independently reproduced. Every row is a different suite; the Decision Index column above is the only shared yardstick.

Read it as a shortlist, not a ranking: a model that scores well on more than one kind of evidence is worth your 200 labelled rows.

Running DecisionBench yourself

DecisionBench evaluates the model's own probability for each candidate answer. It does not parse generated text, so it measures exactly what a decision model returns in production. Its README describes the shortest path: install the package, download the model at a fixed commit, and run a smoke test on the pinned suite.

pip install git+https://github.com/Hanno-Labs/decision-bench.git

# Download an exact model revision, never a moving branch
hf download Hanno-Labs/bosun-v3.1-0.6b \
  --revision aaa9dd06d4d6501b33df61942472fed9284bc5e6 \
  --local-dir models/bosun-v3.1-0.6b

# Smoke test first; drop --smoke for the full run
decision-bench run-hf task_specs/decisionbench-dev.toml \
  models/bosun-v3.1-0.6b results/bosun-v3.1-0.6b --smoke

Check the project's list of supported adapters before you run anything else: if your model is listed, use its runner. Every run writes raw.jsonl, summary.json and manifest.json to its output folder, and the manifest hashes them, so a result can be checked against its inputs later.

Four rules keep your numbers comparable with everyone else's:

1. Pin both revisions. The model at a commit hash, and the dataset at the revision the project publishes for result submissions. A moving branch gives you a number nobody can reproduce.
2. Report the applied suite and the reasoning track separately. DecisionBench 1.0 has a 22,700-row applied suite and a separate 1,200-row reasoning track, and its authors deliberately do not blend them into one headline. Neither should you.
3. Report the whole picture: accuracy, coverage, ECE, NLL, failures (rows the model could not answer or the adapter rejected) and latency, each per view.
4. Keep the raw rows. The aggregate tells you which model is ahead. The rows tell you where it fails, which is what you need before you trust it with your traffic.

If you want your result on the public board, the project's guide to submitting results walks through staging a record against the pinned dataset revision and opening a pull request.

Reading someone else's DecisionBench result

Before you lean on a published DecisionBench score, including the ones on our model pages, ask three questions.

Were the task families in the model's training data? Hanno Labs is unusually clear about this for Bosun: its applied-suite rows are held out, but they come from broad task families that Bosun was trained on, and its authors say that does not establish performance on task families it has never seen. They also report that on the separate reasoning track, Jev leads both Bosun models. That is the pattern to expect from any model that tops a benchmark built by its own team: strong in-distribution, less proven outside it. Your tickets are, by definition, outside it.

What kind of model is it, and what kind is it being compared with? DecisionBench records every result with a declared type: a decision model answers all three primitives through a native decision output; a language model produces text that an adapter parses into an answer; a classifier is a fixed-purpose scorer applied to the benchmark through an adapter. Reviewers check the declared type against how the model is actually served. A native decision model and a prompted chat model can sit on the same board, but their probabilities do not come from the same place, so compare calibration within a type before you compare across types.

Which revision was scored? A result names its model commit and its dataset revision. If the model you would download today is a newer commit, the score describes an older model. Re-run it, or treat the number as history.

Build a 200-row eval from your own data

A public suite cannot know your labels, your edge cases or your language. Two hundred real rows can. The method below takes an afternoon.

1. Sample from production, not from memory. Export the last month or two of tickets, emails or forms. Pick rows at random within each label, so a rare label still gets enough examples, then add every hard case you already know about: refunds disguised as complaints, two requests in one message, other languages.

2. Label them twice. Two people label independently. Discard the rows they disagree on, or better, write down why: a row humans cannot agree on is a row no model should be automated on.

3. Write the question exactly as production will. The same instructions, the same option descriptions, the same order. A reworded question is a different test.

4. Split once, then freeze. Use 100 rows to choose thresholds and fit any calibration, and keep 100 untouched for the final comparison. Never tune on the rows you report.

5. Store it as JSON Lines, one decision per line, in the same format the confidence guide's script reads, so one script computes accuracy, ECE and the risk–coverage curve for every model.

Keep the file local. Real tickets carry names, addresses and account numbers, and a local decision model never needs them to leave your machine.

Compare two models pair by pair

With 200 rows, a two-point accuracy gap is usually noise. The honest comparison runs both models on the same rows and looks only at the rows where they disagree:

# paired.py: rows where exactly one of two models was right
# a_ok, b_ok: lists of booleans, one per row, same order
from scipy.stats import binomtest

def paired(a_ok, b_ok):
    a_only = sum(a and not b for a, b in zip(a_ok, b_ok))
    b_only = sum(b and not a for a, b in zip(a_ok, b_ok))
    n = a_only + b_only
    p = binomtest(a_only, n, 0.5).pvalue if n else 1.0
    print(f"A right, B wrong: {a_only}   B right, A wrong: {b_only}   p = {p:.3f}")

This is the exact form of McNemar's test. Rows both models get right, or both get wrong, say nothing about which is better, so the test ignores them. If the split between the two disagreement counts could easily be chance, treat the models as tied and choose on the other columns: latency, size, licence, calibration. A large p-value with only a handful of disagreements means you need more rows, not a coin toss.

Why one accuracy number is not enough

Two models with the same accuracy can be very different products:

Write the six down for each candidate on your frozen 100 rows, and the decision usually makes itself. If it does not, the cheaper or smaller model wins the tie.

Fit on your hardware: set your GPU or unified memory in the hardware analyzer and this page shows whether each model fits it. Check your hardware →

Sources, checked 2026-10-14: DecisionBench README: install, pinned revision, run-hf · DecisionBench: running the evaluation, run artifacts · DecisionBench: submitting results, pinned dataset revision · Hanno Labs: DecisionBench 1.0 and Bosun v3.1 (applied suite, reasoning track) · DecisionBench: declared model types · Community Decision Index leaderboard · statsmodels: McNemar test · SciPy: binomtest.

Frequently asked questions

Keep going

• Decision models (Jev-style System One): run them locally→
• Run Jev-style decision models locally with Ollama (Nimble, Tev1)→
• Open Jev reproductions beyond Ollama: Winnow, Decider, JevK5, AutoJev, Laya→
• Decision-model confidence, thresholds and calibration — use the probabilities safely→
• Decision models on images: classify receipts, screenshots and forms locally with Clef→
• Decision models on phones, Jetson and laptops→
• Ollama /v1/systemone errors and fixes→
• Ollama /v1/systemone image errors→
• Decision request builder→