Written by Jakub Rusinowski · Last updated June 15, 2026
Last Updated: June 2026 — VRAM (Video RAM) is the single most important factor for running local LLMs. This guide explains exactly how much you need, the math behind it, and which GPUs hit the sweet s
Last Updated: June 2026 — VRAM (Video RAM) is the single most important factor for running local LLMs. This guide explains exactly how much you need, the math behind it, and which GPUs hit the sweet spots.
A model's VRAM requirement is primarily determined by:
VRAM needed ≈ (Parameters × Bytes per parameter) + Context buffer (~2GB)Common quantization formats and their bytes per parameter:
| Format | Bits per Weight | Bytes per Billion Params | Quality vs FP16 |
|---|---|---|---|
| FP16 (no quant) | 16-bit | 2.0 GB | 100% (baseline) |
| Q8_0 | 8-bit | 1.0 GB | ~99% |
| Q4_K_M | 4-bit | 0.5 GB | ~95% |
| Q2_K | 2-bit | 0.25 GB | ~85% |
Example: Llama 3.1 8B at Q4_K_M = (8 × 0.5) + 2 = 6 GB VRAM
| Model | Size | Q4_K_M VRAM | Q8_0 VRAM | Best GPU Tier |
|---|---|---|---|---|
| Qwen 3.5 0.8B | 0.8B | ~1 GB | ~2 GB | Any device (integrated graphics) |
| Qwen 3.5 2B | 2B | ~2 GB | ~3 GB | Any GPU (4GB+), smartphones |
| Gemma 4 E2B | 2B | ~3 GB | ~5 GB | Any GPU (4GB+) |
| Qwen 3.5 4B | 4B | ~3.5 GB | ~6 GB | Any GPU (4GB+) |
| Gemma 3 4B | 4B | 3.8 GB | 6 GB | Any GPU (4GB+) |
| Phi-4 Mini | 3.8B | 4 GB | 5.5 GB | Any GPU (4GB+) |
| Gemma 4 E4B | 4B (Efficient) | ~5.5 GB | ~9 GB | RTX 4060 8GB, Mac M-series |
| Llama 3.1 8B | 8B | 6 GB | 9 GB | RTX 4060 8GB |
| Qwen 3.5 9B | 9B | ~7 GB | ~11 GB | RTX 4060 8GB (tight) / RTX 4060 Ti |
| Cogito v1 8B | 8B | ~5 GB | ~8 GB | RTX 4060 8GB |
| Gemma 3 12B | 12B | 8.1 GB | 13 GB | RTX 4060 Ti / 4070 |
| Qwen 2.5 14B | 14B | 9.5 GB | 15 GB | RTX 4070 12GB |
| Cogito v1 14B | 14B | ~9 GB | ~14 GB | RTX 4070 12GB |
| Devstral-2 22B | 22B | ~13 GB | ~22 GB | RTX 3090 / 4090 |
| Llama 4 Scout | 17B active (MoE) | 10.5 GB | 18 GB | RTX 4080 / 4090 |
| Qwen 3.5 27B | 27B | ~17 GB | ~28 GB | RTX 3090 / 4090 24GB |
| Gemma 3 27B | 27B | 16.5 GB | 28 GB | RTX 4080 / 4090 |
| Gemma 4 26B-A4B | 26B (4B active, MoE) | ~16 GB | ~27 GB | RTX 3090 / 4090 |
| Gemma 4 31B | 31B | ~18 GB | ~31 GB | RTX 3090 / 4090 |
| Nemotron Cascade 2 30B | 30B (Hybrid SSM) | ~17 GB | ~30 GB | RTX 4060 Ti 16GB |
| Cogito v1 32B | 32B | ~20 GB | ~32 GB | RTX 3090 / 4090 24GB |
| DeepSeek R1 32B | 32B | 20 GB | 33 GB | RTX 4090 24GB |
| Llama 4 Maverick | 17B active (MoE) | 24 GB | — | RTX 4090 / Mac M4 Max |
| Qwen 3.5 35B-A3B | 35B (3B active, MoE) | ~20 GB | ~35 GB | RTX 4090 24GB |
| Llama 3.3 70B | 70B | 40 GB | 71 GB | Multi-GPU or Mac Ultra |
| Cogito v1 70B | 70B | ~40 GB | ~70 GB | Dual RTX 3090 / Mac M4 Ultra |
| Nemotron Cascade 2 70B | 70B (Hybrid SSM) | ~40 GB | ~70 GB | Dual RTX 3090 / Mac M4 Ultra |
| Qwen 3.5 122B-A10B | 122B (10B active, MoE) | ~68 GB | — | Multi-GPU cluster |
| GPT-oss 120B | 120B | ~65 GB | — | Dual RTX 3090 / A100 80GB |
| Devstral-2 123B | 123B (40B active, MoE) | ~68 GB | — | Dual-GPU workstation |
| DeepSeek V3.2 685B | 685B (37B active, MoE) | ~370 GB | — | Multi-GPU cluster |
| Qwen 3.5 397B-A17B | 397B (17B active, MoE) | ~220 GB | — | Multi-GPU cluster |
| GLM-5 744B | 744B (40B active, MoE) | ~400 GB | — | Datacenter only |
| Kimi K2.5 1T | 1T+ | ~550 GB | — | API only |
> Tip: MoE (Mixture-of-Experts) models like Llama 4 Scout and Qwen 3.5 35B-A3B use far less VRAM than their total parameter count suggests — only the active parameters run per token.
Apple Silicon (M1/M2/M3/M4) uses unified memory shared between CPU and GPU. This means:
Ollama automatically uses 70% of unified memory as the effective VRAM budget on Apple Silicon.
If the model doesn't fit in VRAM, it spills over to system RAM (CPU offload):
CPU offload is viable for low-latency tasks (e.g., short questions) but frustrating for long documents.
| GPU | VRAM | Price (Est.) | Best Model Tier |
|---|---|---|---|
| RTX 4060 8GB | 8 GB | ~$300 | Llama 3.1 8B, Qwen 3.5 9B, Gemma 4 E4B |
| RX 7600 8GB | 8 GB | ~$250 | Llama 3.1 8B (AMD alternative) |
| RTX 4060 Ti 16GB | 16 GB | ~$450 | Gemma 4 26B-A4B, Nemotron Cascade 2 30B |
| RTX 4070 12GB | 12 GB | ~$550 | Gemma 3 12B, Devstral-2 22B |
| RTX 4070 Ti Super 16GB | 16 GB | ~$750 | Gemma 4 31B, Qwen 3.5 27B |
| RX 7900 XTX 24GB | 24 GB | ~$800 | DeepSeek R1 32B, Qwen 3.5 35B-A3B |
| RTX 4090 24GB | 24 GB | ~$1,600 | Cogito v1 32B, Qwen 3.5 35B-A3B, Llama 4 Maverick |
| RTX 5090 32GB | 32 GB | ~$2,000 | Qwen 3.5 27B Q8, Llama 3.3 70B (partial) |
| Dual RTX 3090 | 48 GB | ~$1,400 | Llama 3.3 70B, Cogito v1 70B |
| Dual RTX 4090 | 48 GB | ~$3,200 | Llama 3.3 70B Q4 at 55+ t/s |
→ Full GPU Buyer's Guide | → Check Your Hardware
How much VRAM for Llama 3 70B? Llama 3 70B at Q4_K_M requires approximately 40–43 GB VRAM. Run it on two RTX 3090s/4090s (48 GB combined) or Apple Silicon with 64 GB+ unified memory.
How does context length affect VRAM usage? Context length increases VRAM via KV cache. At 4k context a 7B model uses ~0.3 GB extra; at 32k it uses ~2.5 GB extra; at 128k it uses ~10 GB extra.
What happens if a model doesn't fit in VRAM? Ollama automatically offloads layers to CPU/RAM. A 70B model on 24 GB GPU runs at 3–8 tokens/sec instead of 30+. Control this with the num_gpu parameter.
Can two GPUs be used together for local LLMs? Yes — Ollama and llama.cpp support multi-GPU via tensor parallelism. Two RTX 3090s give 48 GB VRAM, enough for Llama 3 70B at Q4. NVLink is not required.
Is Q4 quantization good enough quality? Q4_K_M is the community standard — indistinguishable from Q8 for most tasks. Perplexity increases ~0.2–0.5 points vs FP16. Q2 shows noticeable degradation on complex reasoning.