A tokens-per-second figure means nothing without the conditions it was measured under. Record the exact model file, the engine and its version, the context length, the prompt length and the generation length. Then measure prompt processing and generation separately, repeat each run, and report the spread. Do that, and your number can be compared with anyone else's, including the estimates on this site. Skip it, and two people with the same card will disagree by a factor of two and both be right.
What to fix and write down
Every one of these changes the result. Change one at a time, and write all of them down with every run:
| Record | Why it matters |
|---|---|
| Model and exact file: repo, quant (e.g. Q4_K_M), file name or hash | Two "Q4" files of the same model can differ in size and speed; a re-upload can differ from the original |
| Engine and version: llama.cpp build, Ollama version, vLLM release | Engines ship speed changes every few weeks. A number without a version cannot be reproduced next month |
| Context length the model was loaded with, and how full it was | The KV cache grows with context, and generation slows as it fills |
| Prompt length (prefill) | Prompt processing is compute-bound and runs in parallel; it has nothing to do with generation speed |
| Generation length (decode) | Generation is bandwidth-bound and runs one token at a time; this is the "tok/s" most people quote |
| Batch size / concurrency: how many requests at once | One user and sixteen users are different benchmarks; throughput and per-user speed move in opposite directions |
| Sampling: temperature, seed | Mostly affects which tokens come out, but a different length of answer changes the averages |
| GPU offload: layers on GPU, anything on CPU | A model partly in system RAM is measuring your RAM, not your GPU |
| Hardware state: driver, OS, power limit, other programs | A laptop on battery, a power-limited card or a browser playing video all change the result |
| Power draw, if you report efficiency | Tokens per watt is a separate measurement with its own method |
The single most common mistake is reporting one blended number for a whole request. A request that processes a 4,000-token prompt and writes 200 tokens has two very different speeds in it, and averaging them describes neither.
llama.cpp: llama-bench
llama-bench ships with llama.cpp and measures the two phases separately. A pp test processes a prompt of the length you give with -p, and a tg test generates the number of tokens you give with -n. Each test repeats (-r, five times by default), and the result is the mean with its standard deviation, so the spread comes for free. Its README notes that the measurements exclude tokenization and sampling.
# Prompt processing at 512 tokens and generation of 128 tokens, five repetitions each
llama-bench -m ./model-Q4_K_M.gguf -p 512 -n 128 -r 5
# The same test with the cache already filled to 8K, to see how both slow as context grows
llama-bench -m ./model-Q4_K_M.gguf -p 512 -n 128 -d 8192
# Machine-readable output for your notes
llama-bench -m ./model-Q4_K_M.gguf -p 512 -n 128 -o json > run.jsonPin the settings you would actually use: -ngl for how many layers sit on the GPU, -fa for flash attention, -ctk and -ctv for the KV-cache type. Each option takes a comma-separated list, so one command can sweep several values and print them side by side.
Ollama
Ollama's API reports the counts and timings for each request, which is all you need. A /api/generate response includes prompt_eval_count and prompt_eval_duration (the prompt phase) and eval_count and eval_duration (the generation phase), with durations in nanoseconds. Generation speed is eval_count / eval_duration × 10⁹. Prompt speed is the same calculation on the prompt fields.
curl -s http://localhost:11434/api/generate -d '{
"model": "qwen3:8b",
"prompt": "Write a 200-word summary of how a refrigerator works.",
"stream": false,
"options": {"num_ctx": 8192, "seed": 42, "temperature": 0}
}' | python3 -c 'import sys,json; r=json.load(sys.stdin); print("prompt tok/s", r["prompt_eval_count"]/r["prompt_eval_duration"]*1e9, " gen tok/s", r["eval_count"]/r["eval_duration"]*1e9)'Run it once to load the model, throw that run away, then repeat it at least five times and report the median. The first request pays for loading the model from disk (the response's load_duration shows how much).
Serving engines: vLLM, SGLang, TensorRT-LLM
For a server, what you measure is a load pattern, not a single request. Each engine ships a benchmark client:
- vLLM:
vllm bench servesends a dataset of prompts to a running server and reports time to first token, inter-token latency and throughput. Its documentation warns that metric names are not standardised across tools, so compare definitions, not labels. It also warns that repeating a run against the same server can reuse cached prefixes and inflate throughput: vary--seed, restart the server, or usevllm bench sweep serve, which resets caches between runs. For production load testing, the vLLM project points to GuideLLM. - SGLang:
python3 -m sglang.bench_servingis its default load tester. Set--max-concurrency,--num-prompts, and input and output lengths with the random dataset. The docs advise at least five prompts per concurrent slot to reach a steady state. - TensorRT-LLM:
trtllm-benchprepares a synthetic dataset with fixed input and output lengths, then measures throughput against it.
# SGLang: 16 concurrent requests, 256-token prompts, 32-token answers (from its docs)
python3 -m sglang.bench_serving --backend sglang --max-concurrency 16 --num-prompts 80 \
--random-input-len 256 --random-output-len 32 --dataset-name randomReport concurrency with every serving number. A throughput figure at sixteen concurrent requests and a per-user speed at one request are both true, and they describe different machines' worth of experience.
Power draw
To report tokens per watt, sample the board's power during the run, not before or after it. On NVIDIA, nvidia-smi reports the board's average power draw over the last second:
# One reading per second while the benchmark runs in another terminal
nvidia-smi --query-gpu=power.draw --format=csv -l 1 > power.csvAverage the readings from the generation phase only, and say how you measured: a wall-socket meter, the GPU's own sensor and a software estimate give different numbers, and only the first includes the CPU, RAM and power supply.
How this site labels speed figures
Every speed on this site carries a label, so a reader always knows what kind of number it is. The methodology page defines them in full:
- MEASURED: a benchmark run for this exact GPU, model, quant and context, from a named publisher with a source.
- COMMUNITY: the same, where the runs came from readers of this site. The median is treated as authoritative once at least three runs exist for the same cell.
- INTERPOLATED: measured on this GPU for a closely related model, rescaled by the ratio our latency model predicts between the two.
- ESTIMATED: no measurement exists for this pairing. It is the range our calibrated model predicts, always shown as a range.
Most cells on the site are ESTIMATED today. Each reproducible run you submit is what turns one of them into a COMMUNITY figure.
Compare your number with the estimate
Before you submit, open the model's page or the VRAM calculator for your card and compare. The site's ESTIMATED figure is a range for generation speed at a stated quant and context, so compare like with like: your tg result, not a blended request.
- Inside the range: expected. Your run still matters, because it turns a modelled range into a measured point.
- Well below it: check for offloading first (any layers on the CPU, or a model that only just fits), then the power state, then the context: a cache filled to 32K generates more slowly than an empty one.
- Well above it: check that the quant and the model are the ones you think. A smaller quant, or a different checkpoint behind the same name, is the usual explanation. If everything matches, say so in the note. A credible fast result is exactly what recalibrates the estimate.
Submit your result
Speed results go in through the Submit tab on the benchmarks page. It asks for the model, the GPU, the quant, the engine and its version, the context length, the operating system and your generation tok/s, and optionally time to first token and a note. Put the prompt and generation lengths, the repetition count and anything unusual in the note. Nothing is published unreviewed: each submission is checked against the estimate for the same cell, and a large gap means a message, not a silent rejection.
To share your whole machine instead, with what it can run rather than one speed, use Share My Rig.
A one-page checklist
- Load the model once and discard that run.
- Fix context, prompt length, generation length, batch size and seed.
- Measure prompt processing and generation separately.
- Repeat at least five times; report the median and the spread.
- Change one variable at a time.
- Write down the engine version, the exact file and the driver.
- For servers, state the concurrency next to every number.
- Submit through the benchmarks page, with your settings in the note.
Frequently Asked Questions
Why is my tok/s lower than someone else's on the same GPU?
Usually because the two runs measured different things: a different quant file, context length, prompt length, engine version, or one figure blending prompt processing into generation. A partly offloaded model, a power limit or a laptop on battery explains most of the rest. Compare the full list of conditions before you compare numbers.
Should I report prompt processing or generation speed?
Both, separately. Generation speed is what you feel while an answer streams. Prompt processing speed decides how long you wait for the first word after pasting a long document. They are limited by different hardware resources, so one tells you little about the other.
How many runs do I need?
At least five repetitions after a warm-up run, with the median and the spread reported. One run can be skewed by a background task. Five show you whether the number is stable. If the spread is large, find the cause before you publish the median.