Quantization is the process of reducing the precision of the model's weights to save memory (VRAM) and increase speed. Understanding it helps you pick the right trade-off between quality, speed, and hardware requirements.
Quantization: Quality vs VRAM (7B Model)
The Formats
GGUF (GPT-Generated Unified Format)
The current standard for CPU and Apple Silicon inference. It allows models to be split between CPU RAM and GPU VRAM.
- Best for: Mac users, users with low VRAM.
- Tools: Ollama, LM Studio.
EXL2 (ExLlamaV2)
The fastest format for NVIDIA GPUs. It requires the model to fit entirely in VRAM.
- Best for: RTX 3090/4090 owners wanting maximum speed.
AWQ (Activation-aware Weight Quantization)
A robust format supported by many serving engines like vLLM.
Naming Convention (GGUF)
- Q2_K: 2-bit. High perplexity loss. Only use for massive models on small hardware.
- Q3_K_M: 3-bit. Decent for 70B models.
- Q4_K_M: 4-bit. The Sweet Spot. Indistinguishable from FP16 for most tasks.
- Q5_K_M: 5-bit. Slightly better, slightly slower.
- Q6_K / Q8_0: Diminishing returns. Use only if you have VRAM to spare.
Choosing the Right Quantization
| Situation | Recommended Format | Reason |
|---|---|---|
| Plenty of VRAM (model fits easily) | Q8_0 | Near-lossless, fast |
| Tight VRAM (model barely fits) | Q4_K_M | Best quality per GB |
| Model partially spills to CPU | Q4_K_M or Q3_K_M | Minimize layers off-GPU |
| Very limited hardware (RPi, old GPU) | Q2_K or Q3_K_M | Only option that fits |
| Maximum accuracy (research) | FP16 or Q8_0 | No compression artifacts |