Written by Jakub Rusinowski · Last updated June 15, 2026
These are the strongest local models that fit entirely in 6 GB of VRAM, ranked by capability, with the quantization level and estimated tokens/sec needed to fit.
| Qwen 2.5 Family — Qwen 2.5 7B Instruct | Q4_K_M · 4.8 GB · ~93 tok/s on NVIDIA GeForce RTX 5060 |
| Qwen 3.5 (Legacy Listing — Unverified) — Qwen 3.5 7B | Q4_K_M · 4.8 GB · ~93 tok/s on NVIDIA GeForce RTX 5060 |
| IBM Granite 4.1 — Granite 4.1 8B | Q4_K_M · 5 GB · ~90 tok/s on NVIDIA GeForce RTX 5060 |
| StarCoder 2 — StarCoder 2 7B | Q4_K_M · 4.8 GB · ~93 tok/s on NVIDIA GeForce RTX 5060 |
| Gemma 3 — Gemma 3 4B Instruct | Q4_K_M · 3.8 GB · ~118 tok/s on NVIDIA GeForce RTX 5060 |
| Falcon 3 — Falcon 3 7B Instruct | Q4_K_M · 4.8 GB · ~93 tok/s on NVIDIA GeForce RTX 5060 |
| Phi-4 Mini — Phi-4 Mini (3.8B) | Q4_K_M · 2.5 GB · ~179 tok/s on NVIDIA GeForce RTX 5060 |
| Qwen 3.5 — Qwen 3.5 4B | Q4_K_M · 3.4 GB · ~132 tok/s on NVIDIA GeForce RTX 5060 |
| Phi 3.5 Family — Phi 3.5 Mini | Q4_K_M · 2.6 GB · ~172 tok/s on NVIDIA GeForce RTX 5060 |
| Gemma 4 (Legacy Listing — Unverified) — Gemma 4 4B | Q4_K_M · 3.2 GB · ~140 tok/s on NVIDIA GeForce RTX 5060 |
| Cogito v1 — Cogito v1 8B | Q4_K_M · 5 GB · ~90 tok/s on NVIDIA GeForce RTX 5060 |
| OLMo 2 — OLMo 2 7B Instruct | Q4_K_M · 4.5 GB · ~100 tok/s on NVIDIA GeForce RTX 5060 |
| BitNet b1.58 — BitNet b1.58 3B | 1.58-bit · 1.8 GB · ~249 tok/s on NVIDIA GeForce RTX 5060 |
| Gemma 3n — Gemma 3n E4B | Q4_K_M · 3 GB · ~149 tok/s on NVIDIA GeForce RTX 5060 |
| StarCoder 2 — StarCoder 2 3B | Q4_K_M · 2 GB · ~224 tok/s on NVIDIA GeForce RTX 5060 |
Qwen 2.5 Family, Qwen 3.5 (Legacy Listing — Unverified), IBM Granite 4.1, StarCoder 2, Gemma 3 all fit in 6 GB VRAM.
NVIDIA GeForce RTX 5060, NVIDIA GeForce RTX 4060, AMD Radeon RX 9060 XT 8GB, NVIDIA GeForce RTX 5060 Ti 8GB.