Quantization Explained Deep Dive

Written by Jakub Rusinowski · Last updated July 10, 2026

Quantization is the process of reducing the precision of the model's weights to save memory (VRAM) and increase speed. Understanding it helps you pick the right trade-off between quality, speed, and h

In This Guide

• Quantization: Quality vs VRAM (7B Model)
• The Formats
• Naming Convention (GGUF)
• Choosing the Right Quantization

Quantization is the process of reducing the precision of the model's weights to save memory (VRAM) and increase speed. Understanding it helps you pick the right trade-off between quality, speed, and hardware requirements.

Quantization: Quality vs VRAM (7B Model)

The Formats

GGUF (GPT-Generated Unified Format)

The current standard for CPU and Apple Silicon inference. It allows models to be split between CPU RAM and GPU VRAM.

• Best for: Mac users, users with low VRAM.
• Tools: Ollama, LM Studio.

EXL2 (ExLlamaV2)

The fastest format for NVIDIA GPUs. It requires the model to fit entirely in VRAM.

• Best for: RTX 3090/4090 owners wanting maximum speed.

AWQ (Activation-aware Weight Quantization)

A robust format supported by many serving engines like vLLM.

Naming Convention (GGUF)

• Q2_K: 2-bit. High perplexity loss. Only use for massive models on small hardware.
• Q3_K_M: 3-bit. Decent for 70B models.
• Q4_K_M: 4-bit. The Sweet Spot. Indistinguishable from FP16 for most tasks.
• Q5_K_M: 5-bit. Slightly better, slightly slower.
• Q6_K / Q8_0: Diminishing returns. Use only if you have VRAM to spare.

Choosing the Right Quantization

Situation	Recommended Format	Reason
Plenty of VRAM (model fits easily)	Q8_0	Near-lossless, fast
Tight VRAM (model barely fits)	Q4_K_M	Best quality per GB
Model partially spills to CPU	Q4_K_M or Q3_K_M	Minimize layers off-GPU
Very limited hardware (RPi, old GPU)	Q2_K or Q3_K_M	Only option that fits
Maximum accuracy (research)	FP16 or Q8_0	No compression artifacts

← All Guides | Check GPU Compatibility