NVIDIADesktop GPUPrevious generation

NVIDIA GeForce RTX 3060 Ti for local LLMs

Written by Jakub Rusinowski · Last updated

With 8 GB of GDDR6 at 448 GB/s, the RTX 3060 Ti runs 70 catalogued models at Q4_K_M with 8K context. The largest that fits is Jais 3 7B (~7.6 GB), and the top pick is Qwen 3.5 9B at 30–57 tok/s.

8 GB of GDDR6 on a 256-bit bus at 14 Gbps: 448 GB/s, which is more bandwidth than the 12 GB RTX 3060 (360 GB/s) but less memory. NVIDIA also sold an 8 GB GDDR6X version; this record is the GDDR6 card. There is no 12 GB RTX 3060 Ti. Ampere: FP16 and BF16 tensor cores, no FP8.

Models that run on the RTX 3060 Ti

Q4_K_M, 8K context, 8 GB usable. Ranked by quality and speed.

ModelVRAMSpeed
Qwen 3.5 9B
Qwen 3.5
6.6 GB30–57 tok/s
GLM-4 9B
GLM-4.7 / GLM-Z1
6.2 GB26–50 tok/s
GLM-4.6V-Flash 9B
GLM-4.6V
6.2 GB26–50 tok/s
EuroLLM 9B
EuroLLM
6.2 GB26–50 tok/s
MiMo-V2.6 Distill 9B
MiMo-V2.6
6.5 GB30–58 tok/s
Ternary Bonsai 2 9B
Ternary Bonsai 2
6.5 GB30–58 tok/s
InternLM 3 8B Instruct
InternLM 3
6.5 GB30–58 tok/s
LFM2.5-8B-A1B
LFM2.5
5.8 GB71–136 tok/s
Qwen 3 8B
Qwen 3
7 GB28–54 tok/s
Aya Expanse 8B
Aya Expanse
6.7 GB29–56 tok/s
Ministral 8B
Ministral
6.9 GB29–55 tok/s
Gemma 4 E4B
Gemma 4
5.6 GB29–56 tok/s
Showing 12 of 70

Buy it or rent the same memory

Buy the card, or rent a GPU with the same memory by the hour to try models first.

Speed vs other GPUs

Llama 3.1 8B, Q4_K_M. Estimated ranges. How this is calculated

Specifications

Specs last updated 2026-10-07.

Memory
8 GB GDDR6
Memory bandwidth
448 GB/s
Memory bus
256-bit
Architecture
Ampere GA104
Series
RTX 30-series
Board power
200 W
Release year
2020
Launch price
$399
Compute backends
CUDA
Usable for models
8 GB
Best forused 8 GB7-8B modelsbudget CUDA

Similar GPUs

Frequently asked questions

Can the NVIDIA GeForce RTX 3060 Ti run local LLMs?

Yes. With 8 GB (8 GB usable by a model) it runs 70 of the catalogued models at Q4_K_M with 8K context; the largest is Jais 3 7B, needing about 7.6 GB.

How fast is the NVIDIA GeForce RTX 3060 Ti for AI inference?

It is estimated to run Llama 3.1 8B at 32–61 tok/s at Q4_K_M. Llama 3.3 70B does not fit: it needs about 44 GB against 8 GB usable. These are modelled estimates from memory bandwidth, not measurements; the methodology page shows the formula.

What LLMs can I run on 8 GB?

Among the best that fit: Qwen 3.5 9B, GLM-4 9B, GLM-4.6V-Flash 9B, EuroLLM 9B, MiMo-V2.6 Distill 9B.