NVIDIADesktop GPUCurrent

NVIDIA GeForce RTX 4060 for local LLMs

Written by Jakub Rusinowski · Last updated

With 8 GB at 272 GB/s, the RTX 4060 runs 70 catalogued models at Q4_K_M with 8K context. The largest that fits is Jais 3 7B (~7.6 GB), and the top pick is Qwen 3.5 9B at 27–52 tok/s.

Great entry-level AI GPU. 8 GB VRAM is enough for any 7–8B model in Q4 quantization. Only 115W TDP makes it ideal for always-on AI servers.

Models that run on the RTX 4060

Q4_K_M, 8K context, 8 GB usable. Ranked by quality and speed.

ModelVRAMSpeed
Qwen 3.5 9B
Qwen 3.5
6.6 GB27–52 tok/s
GLM-4 9B
GLM-4.7 / GLM-Z1
6.2 GB24–46 tok/s
GLM-4.6V-Flash 9B
GLM-4.6V
6.2 GB24–46 tok/s
EuroLLM 9B
EuroLLM
6.2 GB24–46 tok/s
MiMo-V2.6 Distill 9B
MiMo-V2.6
6.5 GB28–53 tok/s
Ternary Bonsai 2 9B
Ternary Bonsai 2
6.5 GB28–53 tok/s
InternLM 3 8B Instruct
InternLM 3
6.5 GB27–53 tok/s
LFM2.5-8B-A1B
LFM2.5
5.8 GB65–125 tok/s
Qwen 3 8B
Qwen 3
7 GB26–49 tok/s
Aya Expanse 8B
Aya Expanse
6.7 GB27–51 tok/s
Ministral 8B
Ministral
6.9 GB26–50 tok/s
Gemma 4 E4B
Gemma 4
5.6 GB26–51 tok/s
Showing 12 of 70

Buy it or rent the same memory

Buy the card, or rent a GPU with the same memory by the hour to try models first.

Affiliate disclosure: Some links on this page are affiliate links — if you buy through them, LLM Configurator may earn a commission at no extra cost to you. As an Amazon Associate, LLM Configurator earns from qualifying purchases.
NVIDIA GeForce RTX 4060 8GB
8 GB VRAM · 115 W board power
2026 prices are volatile — check the current listing.

Speed vs other GPUs

Llama 3.1 8B, Q4_K_M. Estimated ranges. How this is calculated

Specifications

Specs last updated 2026-07-12.

Memory
8 GB
Memory bandwidth
272 GB/s
Architecture
Ada Lovelace AD107
Series
RTX 40-series
Board power
115 W
Release year
2023
Launch price
$299
Usable for models
8 GB
Best for8B modelsbudgetlow power AIbeginners
Recommended first model (Ollama)

Similar GPUs

Frequently asked questions

Can the NVIDIA GeForce RTX 4060 run local LLMs?

Yes. With 8 GB (8 GB usable by a model) it runs 70 of the catalogued models at Q4_K_M with 8K context; the largest is Jais 3 7B, needing about 7.6 GB.

How fast is the NVIDIA GeForce RTX 4060 for AI inference?

It is estimated to run Llama 3.1 8B at 29–56 tok/s at Q4_K_M. Llama 3.3 70B does not fit: it needs about 44 GB against 8 GB usable. These are modelled estimates from memory bandwidth, not measurements; the methodology page shows the formula.

What LLMs can I run on 8 GB?

Among the best that fit: Qwen 3.5 9B, GLM-4 9B, GLM-4.6V-Flash 9B, EuroLLM 9B, MiMo-V2.6 Distill 9B. The quickest start is Ollama: ollama run llama3.1:8b.