本地大模型速度：按 GPU 与模型
选择你的显卡和模型，查看它在你的上下文长度下实际能跑多快，并附上显存计算过程。估算以区间呈现，实测数据注明来源。
这些数字是如何得出的
Decode speed for all 220 runnable local LLM variants across 109 GPUs. Per-token latency is modelled as three terms — streaming the active weights and KV cache from memory, any penalty for weights that spill to system RAM, and a fixed framework floor — with efficiency fitted per GPU architecture from 14 published llama-bench measurements (mean absolute error 7.68%). Fastest models on an NVIDIA RTX 4090 (24 GB) at 4K context:
| SmolLM2 (360M, Q4_K_M) | 257–495 tok/s (modelled range) |
| Ternary Bonsai 2 (2B, PTQ1_0 ternary) | 249–479 tok/s (modelled range) |
| Qwen 3.5 (0.8B, Q4_K_M) | 224–431 tok/s (modelled range) |
| DeepSeek-OCR (3B (0.57B active), Q4_K_M) | 224–429 tok/s (modelled range) |
| Llama 3.2 Family (1B, Q4_K_M) | 215–412 tok/s (modelled range) |
| Gemma 3 (1B, Q4_K_M) | 213–409 tok/s (modelled range) |
| Ternary Bonsai 2 (4B, PTQ1_0 ternary) | 202–388 tok/s (modelled range) |
| BitNet b1.58 (3B, 1.58-bit) | 201–385 tok/s (modelled range) |
| MiniCPM-V (1.3B, Q4_K_M) | 200–384 tok/s (modelled range) |
| VibeThinker (1.5B, Q4_K_M) | 190–365 tok/s (modelled range) |
| OLMoE (7B (1.3B active), Q4_K_M) | 188–360 tok/s (modelled range) |
| LFM2.5 (8.3B (1.5B active), Q4_K_M) | 178–342 tok/s (modelled range) |
| LFM2.5 (24B (2.45B active), Q4_K_M) | 175–336 tok/s (modelled range) |
| Mellum2.1 (12B (2.5B active), Q4_K_M) | 174–335 tok/s (modelled range) |
| Qwen 3.5 (2B, Q4_K_M) | 174–334 tok/s (modelled range) |
| Gemma 3n (2B, Q4_K_M) | 170–327 tok/s (modelled range) |
| EXAONE 3.5 (2.4B, Q4_K_M) | 166–318 tok/s (modelled range) |
| SmolLM2 (1.7B, Q4_K_M) | 163–312 tok/s (modelled range) |
| Trinity (26B (3B active), Q4_K_M) | 161–309 tok/s (modelled range) |
| Qwen 3.6 (35B (3B active), Q4_K_M) | 160–308 tok/s (modelled range) |
or compare on Vast.ai from $0.35/hr (typical low · varies)
As an Amazon Associate we earn from qualifying purchases. Cloud GPU links are referral links — we may earn a commission at no extra cost to you.
Modelled figures are shown as ranges rather than single numbers, because the model's accuracy does not justify more precision than that. Rows marked measured come from published benchmarks credited to their publisher with a source URL, or from reader submissions once several runs agree. Speed depends on context length: the same 8B model loses roughly half its throughput between 4K and 65K context, because the KV cache re-read on every token grows until it dwarfs the weights. Mixture-of-Experts models are sized by their TOTAL parameters for the memory-fit check even though only the active experts affect speed, so a 109B model with 17B active will not load on a 24 GB card. The open dataset is published under CC BY 4.0 at /measured-benchmarks.json.
Comparing the hardware itself? Every device we track — memory, bandwidth and $/GB side by side — or narrow it to GPUs, laptops or AI stations.
查看这里最常被比较的显卡价格
每个显存档位一张卡，与我们的 GPU 选购指南一致。购买前请先在上方表格中查看你的模型。