Methodology

Written by Jakub Rusinowski · Last updated August 17, 2026

Every number on this site is either a measurement with a source or the output of one of the two formulas below. This page publishes both, the constants they use, and the cases where we know the model is wrong.

VRAM uses TOTAL parameters; speed uses ACTIVE parameters. For a dense model these are the same number, which is why conflating them goes unnoticed. For a Mixture-of-Experts model they diverge by up to 20x. Every expert must be resident in memory because the router can select any of them on any token, so VRAM is a function of TOTAL parameters. Only the routed experts are read per token, so throughput is a function of ACTIVE parameters. Llama 4 Scout is 17B active within 109B total: it needs 65.8 GB of VRAM at Q4_K_M but streams only 10.3 GB per token, which is why it can decode like a 17B model on hardware large enough to hold it.

1. VRAM

What must be resident in memory for the model to load at all.

vram_gb = weights_gb + kv_cache_gb + overhead_gb

weights_gb   = totalParamsB x bits_per_weight / 8      <- TOTAL parameters
kv_cache_gb  = 2 x layers x kv_heads x head_dim x ctx x kv_bytes / 1e9
overhead_gb  = 0.8                                    framework + CUDA context

Bits per weight

Effective on-disk width including block scales and metadata, which is why the 4-bit formats land above 4.0.

Quantization	Bits per weight
`Q2_K`	2.63
`Q3_K_M`	3.41
`Q4_K_M`	4.83
`Q5_K_M`	5.67
`Q6_K`	6.56
`Q8_0`	8.50
`F16`	16.00

2. Decode throughput

Tokens per second during generation. Three physically distinct terms, not a single bandwidth division.

t_token_ms = (W_active + KV(ctx)) / (BW × η_arch) × 1000     memory streaming
           + W_offloaded / (BW_host × η_host)     × 1000     offload penalty
           + t_fixed_ms                                       framework floor

tokens_per_sec = 1000 / t_token_ms

Term	What it is	Why it is there
`W_active`	Quantized bytes of the weights actually read per token.	For a dense model this is the whole file. For a Mixture-of-Experts model it is only the routed experts, which is why a 109B MoE can decode as fast as a 17B dense model. Note this is the SPEED input only — the FIT check uses total weights, because every expert must be resident.
`KV(ctx)`	2 × n_layers × n_kv_heads × head_dim × ctx × bytes_per_element, from each model's published config.json.	The KV cache is re-read every token, so it adds to the streaming cost and grows linearly with context. This is the single biggest omission in a plain roofline model: an 8B Q4 loses roughly half its throughput between 4K and 65K context purely because of this term. Models using Multi-head Latent Attention (DeepSeek, Kimi) cache one compressed latent instead of K and V, which is 10-20× smaller, and are computed separately.
`η_arch`	Fraction of theoretical memory bandwidth a decode loop actually sustains, fitted per GPU architecture.	Sustained bandwidth is a property of the memory subsystem, not of the model being run. Ada and Ampere sit about 0.30 apart in our fit, so a single global constant would misattribute an architectural difference to every model on the page.
`t_fixed`	Per-token latency that does not scale with model size: kernel launch, graph dispatch, sampling. Fitted per backend.	This is why a 0.5B model does not run 40× faster than a 20B one on the same card. It also replaces the hardcoded 400 tok/s ceiling this page used to carry: throughput now approaches 1000/t_fixed as weights shrink, so the ceiling is a consequence of the backend rather than a number someone chose.
`W_offloaded / BW_host`	Weights that do not fit in VRAM, read from system RAM by the CPU.	llama.cpp does not stream non-resident weights to the GPU across PCIe — it computes those layers on the CPU, reading them from system DRAM, and only the activation tensor crosses the link. So the bottleneck is host memory bandwidth, not link bandwidth. On unified-memory systems (Apple, DGX Spark, Strix Halo) there is no second tier at all and this term is zero.

3. Memory-bandwidth utilization

A decode step is memory-bound: the weights that are read per token have to cross the memory bus, and no real runtime sustains 100% of a card's theoretical peak. Assuming 1.0 would overstate throughput by 25-40%. This site fits a per-architecture efficiency to published llama-bench runs rather than assuming a single figure, because sustained bandwidth is a property of the memory subsystem and not of LLMs. The fitted values are in the table below. End-to-end, after the fixed framework cost is included, the effective utilization these produce lands between roughly 0.45 and 0.85 depending on card and model size.

GPU architecture	η	Fitted?	Basis
NVIDIA Blackwell	0.850	seed assumption	assumption — no calibration data for this architecture
NVIDIA Ada Lovelace	0.927	fitted	solved from 12 measurements
NVIDIA Ampere	0.622	fitted	solved from 1 measurement
NVIDIA Hopper	0.850	seed assumption	assumption — no calibration data for this architecture
AMD RDNA 4	0.700	seed assumption	assumption — no calibration data for this architecture
AMD RDNA 3 / 3.5	0.650	seed assumption	assumption — no calibration data for this architecture
Intel Xe2 (Battlemage)	0.600	seed assumption	assumption — no calibration data for this architecture
Apple Silicon	0.548	fitted	solved from 1 measurement
Custom hardware (unclassified)	0.750	seed assumption	assumption — no calibration data for this architecture
CPU (system RAM)	0.350	seed assumption	assumption — no calibration data for this architecture

4. The small-model ceiling

Kernel launch, graph dispatch and sampling cost the same whether the model is 0.5B or 70B. As the weights shrink, that fixed cost comes to dominate and throughput asymptotes to 1000/t_fixed rather than rising without limit. This site models that as a measured per-backend constant (t_fixed) instead of a hardcoded ceiling, so the flattening is an emergent property of the backend and different small models still return different numbers. A hard clamp was removed precisely because it made a 1B model and a 109B model report the same value, and a test fails the build if one reappears.

Backend	t_fixed (ms)	Implied ceiling (tok/s)	Fitted?
CUDA	2.25 ms	—	fitted
ROCM	3.50 ms	—	seed assumption
METAL	3.00 ms	—	seed assumption
VULKAN	4.00 ms	—	seed assumption
CPU	5.00 ms	—	seed assumption

5. How good is it?

Fitted against 14 published llama-bench runs. Mean absolute percentage error 7.68% (10.68% counting only rows whose exact checkpoint is confirmed). Worst single residual 31.53%. 3 of 10 GPU architectures have measurements behind them; the rest carry seed values and are flagged as uncalibrated wherever they appear.

6. Confidence labels

No number is published bare. Every throughput figure carries one of these:

Label	Meaning
MEASURED	From verified benchmark submissions for this exact GPU, model, quantization and context. Shown with its spread and sample count.
COMMUNITY	A measured value whose runs all came from readers of this site, shown with its sample count.
ESTIMATED	No measurement exists for this pair. The range is what the calibrated latency model predicts, and it is shown as a range because a single number would imply precision the model does not have.
INTERPOLATED	Measured on this GPU for a closely related model, rescaled by the ratio the latency model predicts between them.

7. Known limitations

Published because a reader who finds these independently trusts the rest of the page less than one who was told.

Small-active-parameter MoE at long context is over-predictedQwen3-30B-A3B at 57K context on an RTX 4090 predicts about 98 tok/s against 74.6 measured, a 31% error — the single worst point in the calibration set and the only one above 25%. The same model at 4K is within 11%, and the dense Qwen3-8B series holds within 5% out to 128K, so this is not a general long-context failure. Likely causes are expert-routed reads sustaining less bandwidth than sequential dense reads, and the card running near capacity. With only two MoE points in the set, adding a term to correct it would be fitting a parameter to one observation.

Apple Silicon rests on a single measurement, and two sources disagreeAn M3 Max at 400 GB/s measured at 40 tok/s and an M4 Max at 546 GB/s published at 75-90 tok/s imply a 1.9-2.25× speedup from a 1.37× bandwidth increase. No single Apple efficiency constant satisfies both. We fit to the M3 Max point, which means the M4 Max held-out check fails — our prediction of about 52 tok/s falls below the published range. Either one source is wrong or Apple generations need separate constants, and one measurement cannot settle it.

The offload path is not calibrated at allEvery published llama-bench figure we could source runs fully resident at -ngl 99, so no measurement constrains the offload term. Its constants (89.6 GB/s host DRAM at 35% efficiency) are assumptions, and any row whose weights spill to system RAM is reported as uncalibrated with a wide band. Treat offloaded figures as order-of-magnitude only.

Most architectures have no measurements behind themBlackwell, Hopper, RDNA 3, RDNA 4 and Intel Xe2 have no calibration rows. Their efficiency constants are seeded assumptions, flagged as unfitted throughout the site, and estimates on those cards carry a deliberately wider band. Bandwidth ratios between cards of the same architecture should still be reliable; absolute figures on an unfitted architecture should not be trusted to better than about a third.

Roughly half the model library has no verified architecture72 of 154 library variants have a config.json we could verify, and their KV figures are exact. The rest are unreleased, speculative, or explicitly unverified listings, and their KV cache is inferred from parameter count by a documented heuristic. Those rows report their KV provenance as estimated rather than measured.

The residency rule is slightly too conservative at very long contextQwen3-8B Q4_K_M at 128K context on a 24 GB RTX 4090 computes to 25.1 GB required — weights plus a 19.3 GB KV cache plus framework overhead — so we predict it will not load. Hardware Corner measured it running at 33.8 tok/s. Flash attention and a smaller real KV footprint close a roughly 4% gap our arithmetic does not model. Where a measurement exists past a predicted memory cliff, the page shows it and says the rule is wrong rather than hiding the point; trust the measurement.

Prompt processing is calibrated on one architecture onlyPrefill (and therefore time-to-first-token) is modelled as a multiple of decode speed: prefill_tok_s = decode_tok_s x R_arch x (active/total params). R is solved from three published prompt-processing figures, all on an RTX 4090, giving 72.8 with a 5.4% mean error. Every other architecture inherits that value and reports uncalibrated, so TTFT off Ada is a rough guide rather than an estimate. Apple Silicon is the weakest case: it has far less compute per unit of bandwidth than a discrete NVIDIA part, so its true prefill advantage is almost certainly smaller than the figure shown.

Four calibration rows do not name their exact checkpointSome sources report only a size class ("7B", "14B"). Those rows are fitted against a same-geometry stand-in and tagged as inferred identity, and the fit report gives MAPE both with and without them so the headline figure can be read either way.

Benchmarks | VRAM calculator | Check your hardware