Local LLM speed, by GPU and model
Pick your card and your model to see how fast it will actually run — at your context length, with the memory maths shown. Estimates are ranges, measurements are credited to their source.
How these numbers are produced
Decode speed for all 220 runnable local LLM variants across 109 GPUs. Per-token latency is modelled as three terms — streaming the active weights and KV cache from memory, any penalty for weights that spill to system RAM, and a fixed framework floor — with efficiency fitted per GPU architecture from 14 published llama-bench measurements (mean absolute error 7.68%). Fastest models on an NVIDIA RTX 4090 (24 GB) at 4K context:
| SmolLM2 (360M, Q4_K_M) | 257–495 tok/s (modelled range) |
| Ternary Bonsai 2 (2B, PTQ1_0 ternary) | 249–479 tok/s (modelled range) |
| Qwen 3.5 (0.8B, Q4_K_M) | 224–431 tok/s (modelled range) |
| DeepSeek-OCR (3B (0.57B active), Q4_K_M) | 224–429 tok/s (modelled range) |
| Llama 3.2 Family (1B, Q4_K_M) | 215–412 tok/s (modelled range) |
| Gemma 3 (1B, Q4_K_M) | 213–409 tok/s (modelled range) |
| Ternary Bonsai 2 (4B, PTQ1_0 ternary) | 202–388 tok/s (modelled range) |
| BitNet b1.58 (3B, 1.58-bit) | 201–385 tok/s (modelled range) |
| MiniCPM-V (1.3B, Q4_K_M) | 200–384 tok/s (modelled range) |
| VibeThinker (1.5B, Q4_K_M) | 190–365 tok/s (modelled range) |
| OLMoE (7B (1.3B active), Q4_K_M) | 188–360 tok/s (modelled range) |
| LFM2.5 (8.3B (1.5B active), Q4_K_M) | 178–342 tok/s (modelled range) |
| LFM2.5 (24B (2.45B active), Q4_K_M) | 175–336 tok/s (modelled range) |
| Mellum2.1 (12B (2.5B active), Q4_K_M) | 174–335 tok/s (modelled range) |
| Qwen 3.5 (2B, Q4_K_M) | 174–334 tok/s (modelled range) |
| Gemma 3n (2B, Q4_K_M) | 170–327 tok/s (modelled range) |
| EXAONE 3.5 (2.4B, Q4_K_M) | 166–318 tok/s (modelled range) |
| SmolLM2 (1.7B, Q4_K_M) | 163–312 tok/s (modelled range) |
| Trinity (26B (3B active), Q4_K_M) | 161–309 tok/s (modelled range) |
| Qwen 3.6 (35B (3B active), Q4_K_M) | 160–308 tok/s (modelled range) |
or compare on Vast.ai from $0.35/hr (typical low · varies)
As an Amazon Associate we earn from qualifying purchases. Cloud GPU links are referral links — we may earn a commission at no extra cost to you.
Modelled figures are shown as ranges rather than single numbers, because the model's accuracy does not justify more precision than that. Rows marked measured come from published benchmarks credited to their publisher with a source URL, or from reader submissions once several runs agree. Speed depends on context length: the same 8B model loses roughly half its throughput between 4K and 65K context, because the KV cache re-read on every token grows until it dwarfs the weights. Mixture-of-Experts models are sized by their TOTAL parameters for the memory-fit check even though only the active experts affect speed, so a 109B model with 17B active will not load on a 24 GB card. The open dataset is published under CC BY 4.0 at /measured-benchmarks.json.
Comparing the hardware itself? Every device we track — memory, bandwidth and $/GB side by side — or narrow it to GPUs, laptops or AI stations.