LLM Quantization Explained: How a 70B Model Shrinks from 140 GB to 8 GB

One of the most common questions from people new to local LLMs: "How can I run a 70 billion parameter model on a single consumer GPU with 24 GB of VRAM? Doesn't a 70B model weigh hundreds of gigabytes?"

The answer is quantization — one of the most important practical techniques in the local AI ecosystem, and one that most users interact with constantly without fully understanding what's happening under the hood.

This guide explains what quantization is, how different formats and bit depths work, what quality you trade away, and exactly which quantization level to use for your hardware and use case.

The Raw Numbers Problem

To understand quantization, start with the math of uncompressed model weights.

Every neural network parameter — each "weight" in the model — is a floating-point number that encodes something the model learned during training. In standard machine learning, weights are stored as 32-bit floating point (FP32): 4 bytes per number.

A 70 billion parameter model at FP32: 70,000,000,000 × 4 bytes = 280 GB

That's just the weights. Inference also requires activations (the intermediate values computed during a forward pass), the KV cache for context, and overhead — easily pushing total memory requirements past 300 GB for long contexts.

No consumer GPU comes close to 300 GB of VRAM. The highest-end single consumer card as of 2026 — the RTX 5090 — has 32 GB. Professional cards top out at 80 GB (H200).

This is where quantization enters.

What Quantization Actually Does

Quantization reduces the numerical precision used to represent each weight. Instead of a 32-bit float (which can represent any value between roughly ±3.4 × 10³⁸ with high precision), you use fewer bits — accepting reduced precision in exchange for dramatically smaller memory footprint.

The intuition: most model weights don't need 32-bit precision. Training uses FP32 to preserve gradient accuracy during the learning process. But once training is complete, the weights are essentially fixed lookup values. For inference, a weight of 0.732441902... doesn't need to be distinguished from 0.732. The prediction quality degrades far less than you'd expect.

Floating Point Formats

FP32 (4 bytes/weight): Full precision. Used during training. Too large for consumer inference of large models. BF16 (2 bytes/weight): 16-bit "brain float". Same range as FP32 but half the precision. Used for inference on cloud hardware. 70B model = 140 GB. Still too large for most consumer GPUs, but runs well on dual-GPU setups. FP16 (2 bytes/weight): 16-bit IEEE float. Similar to BF16, used natively on NVIDIA CUDA. 70B = 140 GB.

Integer Quantization

The real compression comes from integer quantization: representing weights as integers rather than floats.

INT8 (1 byte/weight): 256 possible values per weight. 70B model = 70 GB. Quality loss is minimal for most tasks. Runs on single high-VRAM consumer GPU (RTX 6000 Ada, 48 GB) or dual RTX 4090s. INT4 (0.5 bytes/weight): 16 possible values per weight. 70B model = ~35 GB. Small but perceptible quality loss depending on implementation. Single RTX 4090 (24 GB) can't quite hold the full model + KV cache, but 40+ GB setups work. INT3 (~0.375 bytes/weight): 8 possible values. 70B model = ~26 GB. Noticeable quality degradation. Rarely used in production.

GGUF: The Format That Made Local AI Accessible

The format you'll encounter most often for local LLMs is GGUF (GPT-Generated Unified Format), developed by Georgi Gerganov, the creator of llama.cpp. It replaced the older GGML format in August 2023 and has become the standard for distributable quantized models.

GGUF stores model weights, tokenizer data, and metadata in a single portable file. More importantly, it supports mixed-precision quantization — different parts of the model can use different bit depths, preserving quality where it matters most.

When you download a model from Hugging Face for use with Ollama, LM Studio, or llama.cpp, you're almost certainly downloading a GGUF file.

The Q-Prefix System

GGUF models use a naming convention that tells you the quantization level:

The _K suffix means the model uses "k-quant" grouping — weights are quantized in groups, with scale factors preserved at higher precision. This significantly improves quality over naive quantization at the same bit depth. Always prefer _K variants when available.

The _M (medium) and _S (small) suffixes indicate which layers of the model get higher vs. lower precision. _M variants give higher precision to the most sensitive layers (embedding and attention layers), producing better quality for a small size increase over _S.

How Much Quality Do You Actually Lose?

This is the practical question that matters. The answer depends on the task and the quantization level.

Benchmark Comparison: Llama 3.3 70B at Different Quantizations

The following data represents typical quality degradation measured against FP16 baseline (100%):

The practical takeaway: Q4_K_M is the sweet spot for most users. It retains ~98% of the model's benchmark quality while reducing a 70B model from 140 GB to ~37 GB. On creative writing, summarization, coding, and most Q&A tasks, Q4_K_M and Q5_K_M are indistinguishable to human evaluators in blind tests.

Quality loss becomes perceptible at Q3 and significant at Q2. If you find yourself using Q2 to squeeze a model onto your hardware, you're usually better off using a smaller, higher-quality model at Q4 or Q5.

Smaller Models, Better Quality: The Practical Tradeoff

This is the insight that changes hardware purchasing decisions: a small model at high quantization often beats a large model at severe quantization.

Consider the choice for a developer with 24 GB VRAM (RTX 4090):

Option D — Qwen 3.5 32B at Q5_K_M — is almost certainly the best choice for a 24 GB GPU user. You get a capable 32B model at near-lossless quantization, fitting comfortably with room for KV cache.

Option A (70B at Q2_K) sounds impressive but the aggressive compression strips away enough precision that a well-quantized 14B model often outperforms it on specific reasoning tasks.

Newer Techniques: Beyond Standard Integer Quantization

The GGUF/GGML ecosystem has continued to innovate. Several newer approaches improve on basic integer quantization:

GPTQ (Gradient-based Post-Training Quantization)

GPTQ uses information about the model's training data to make smarter decisions about which weights to quantize aggressively vs. which to preserve. It runs a calibration pass on a representative dataset after training, then applies per-weight optimization to minimize reconstruction error.

GPTQ 4-bit models typically outperform standard Q4 models on most benchmarks by 0.5–1.5 percentage points. They require slightly more preprocessing time but produce better results. GPTQ models are available for most major models on Hugging Face.

AWQ (Activation-aware Weight Quantization)

AWQ improves on GPTQ by observing that not all weights are equally important — weights that receive large activations are more sensitive to precision loss. AWQ selectively protects these high-impact weights while aggressively quantizing lower-impact ones.

In practice, AWQ 4-bit models often match Q5 GGUF quality while using Q4 memory footprint. Available through llama.cpp's AWQ support and vLLM.

EXL2 (ExLlamaV2 Format)

EXL2 takes the per-weight importance idea further — you can specify an average bits-per-weight target (e.g., 3.5 bits) and the algorithm distributes precision across weights to minimize quality loss at that target. This produces models that are more customized for specific quality/size tradeoffs than the fixed GGUF Q-levels.

EXL2 models are primarily used with the ExLlamaV2 inference engine, which has excellent NVIDIA GPU performance (often 20–30% faster than llama.cpp on compatible hardware).

Choosing the Right Quantization for Your Hardware

Here's a practical decision framework:

Apple Silicon (M3/M4 Pro, 36–48 GB unified memory)

Target: Q6_K or Q8_0 for 14B–32B models. The unified memory architecture is efficient at higher precision, and you have enough memory to run Q6_K on most models without hitting the ceiling.

# Best quality for M4 Pro 36GB
ollama pull qwen3.5:32b-q6_k    # ~33 GB, near-lossless
ollama pull llama3.3:70b-q3_k_m # ~24 GB, fits with some headroom

RTX 4090 (24 GB VRAM)

Target: Q4_K_M or Q5_K_M for 30B–32B models. This is the sweet spot — Q4_K_M gives you models up to ~32B at full quality, or 70B at Q2 (not recommended).

# Best choice for 24 GB GPU
ollama pull qwen3.5:32b-q4_k_m  # ~19 GB, excellent quality
ollama pull qwen3.5:32b-q5_k_m  # ~22 GB, near-lossless

RTX 4080 / RX 7900 XTX (16 GB VRAM)

Target: Q4_K_M or Q5_K_M for 14B models. At 16 GB, you're limited to 7B–14B models at good quality, or 32B at heavy compression (not recommended).

# Best choice for 16 GB GPU
ollama pull qwen3.5:14b-q5_k_m  # ~10 GB, near-lossless
ollama pull mistral-small3.1:22b-q3_k_m  # ~10 GB, decent quality

RTX 3060 / RX 7600 (8 GB–12 GB VRAM)

At 8–12 GB, you're running 7B–8B models. Use Q6_K or Q8_0 — you have the headroom for high quality at this model size.

# Best choice for 8-12 GB GPU
ollama pull llama3.2:8b-q8_0    # ~8.5 GB, near-lossless
ollama pull gemma3:9b-q6_k      # ~7 GB, near-lossless

CPU Offloading: When You Don't Have Enough VRAM

If your model doesn't fully fit in VRAM, llama.cpp and Ollama support CPU offloading — some transformer layers run on the GPU, the rest on CPU via RAM.

The performance hit is severe: each layer on CPU instead of GPU reduces throughput by roughly 30–40%. A 70B Q4 model split across 24 GB GPU + 16 GB RAM might run at 2–5 tokens/second instead of the 20–35 t/s you'd get fully on-GPU.

For interactive chat, 2–5 t/s is usable but slow. For batch processing where latency doesn't matter, it's viable.

# Ollama: manually set number of GPU layers (rest offload to CPU)
OLLAMA_GPU_LAYERS=40 ollama run llama3.3:70b-q4_k_m

# llama.cpp: equivalent flag
./llama-cli -m llama3.3-70b-q4_k_m.gguf -ngl 40  # 40 layers on GPU

The Bottom Line

Quantization is why local LLMs exist as a consumer product. Without it, running any model larger than 7B would require enterprise hardware. With Q4_K_M quantization, a 70B model fits in 37 GB — a single workstation GPU. With Q5_K_M, a 32B model fits in 22 GB — a standard gaming card.

The practical rules:

1. Never go below Q4 if you can avoid it. The quality loss at Q3 and Q2 is real and observable. 2. Match model size to quantization: A well-quantized 14B model beats a poorly-quantized 70B model. 3. Q4_K_M is the universal sweet spot: Best quality-per-GB for most use cases. 4. Q8_0 if you have the VRAM: The quality delta from Q8 to FP16 is almost unmeasurable. 5. Consider GPTQ or AWQ for maximum quality at a given bit depth, especially for coding tasks.

The story of local LLMs is largely the story of quantization research. Every improvement in this area directly translates to better models running on cheaper hardware — and that trajectory is nowhere near its end.

---

Compare models and their quantized sizes → Model Library Find out which quantized models fit your GPU → Hardware Analyzer