How Much VRAM Do You Need for Local LLMs?

Short answer: VRAM needed ≈ parameters × bits per weight ÷ 8, plus the KV cache (which grows with context) and about 0.8 GB of overhead. At Q4_K_M that is roughly 0.6 GB per billion parameters, so an 8B model needs about 7 GB at 8K context, a 32B model about 23 GB, and a 70B model about 46 GB. The table below is computed for a range of models and quantisations. Find your model, read across, and compare with your card.

VRAM needed by model size and quantisation

VRAM in GB, computed by our VRAM engine: weights + KV cache at 8K context + framework overhead. For Mixture-of-Experts models the Parameters column shows both numbers on purpose. VRAM follows the total (every expert must be resident, because the router can pick any of them on any token). Speed follows the active count (only the chosen experts are read per token). That is why Qwen 3 30B-A3B needs the memory of a 30B model but streams like a 3B one, and why Llama 4 Scout needs a server-class machine despite its 17B active parameters.

The formula

Weights take parameters × bits per weight ÷ 8 bytes. The bits per weight are not the nominal ones, because quantised formats carry block metadata:

Weights only. Add the KV cache (grows with context) and about 0.8 GB of framework overhead. Bits per weight include quantisation block metadata, so Q4_K_M is 4.83 bits, not 4. The KV cache is the memory a model uses to remember the conversation. It grows linearly with context length and depends on the architecture: models with few KV heads use much less, DeepSeek and Kimi-style models use a compressed latent cache, and hybrid models such as Qwen3-Next keep a cache in only some layers (12 of its 48). llama.cpp can also store the cache at lower precision: q8_0 roughly halves it, and quantising the V cache requires flash attention to be enabled (llama.cpp discussion, error explainer).

What fits in your VRAM

The largest models that fit each memory size, by the same calculation, without spilling into system RAM:

Sized by the same engine: total parameters at Q4_K_M, 8K context, never offloading to system RAM.

Context length and defaults

The tables assume 8K of context. Ollama's default context is set by your VRAM: 4K below 24 GiB, 32K from 24 to 48 GiB, and 256K at 48 GiB or more (Ollama docs, Fast.io). A larger context needs more memory, so a model that fits at 8K can fail to load at 128K. Set the context explicitly when you size a model.

Apple Silicon: unified memory

Apple Silicon shares one memory pool between the CPU and GPU, so a 64 GB Mac can hold models a 24 GB graphics card cannot. macOS does not give the GPU all of it by default: the cap is roughly two-thirds of RAM on machines up to 36 GB and about three-quarters above that (ModelPiper, Apple Silicon LLM guide). You can raise it with the iogpu.wired_limit_mb setting. Memory bandwidth is lower than on a fast discrete GPU, so tokens per second are lower for the same model.

What happens when it does not fit

If a model does not fit, the runtime offloads some layers to system RAM. It still runs, but decode speed falls to a fraction of GPU speed, and the more layers offloaded, the slower it gets. Use the GPU compatibility checker to see the fit verdict for your exact card, model and context before you download.

If you keep hitting the limit, the next step is more VRAM. See the GPU buyer's guide for cards by tier, with launch MSRP and dated street price.

Frequently Asked Questions

How much VRAM do I need to run a 70B model?

About 46 GB at Q4_K_M with 8K of context, so two 24 GB cards, a 48 GB workstation card, or a Mac with 64 GB or more of unified memory. A single 24 GB card cannot hold it.

How does context length affect VRAM?

Through the KV cache, which grows linearly with the number of tokens. The size depends on the model's architecture, so use the calculator rather than a fixed rule. Quantising the KV cache to q8_0 roughly halves it.

What happens if a model does not fit in VRAM?

The runtime offloads layers to system RAM and generation slows down substantially. It works for occasional use but is frustrating for long documents.

Can two GPUs be used together for local LLMs?

Yes. llama.cpp and Ollama can split a model across several GPUs, so two 24 GB cards give you a 48 GB pool. Bandwidth between cards affects speed, but NVLink is not required.

Is Q4 quantisation good enough?

Q4_K_M is the common default and holds up well for most tasks, but it is a trade: higher-bit formats such as Q5_K_M or Q8_0 lose less quality and cost more memory. Very low-bit formats show clearer degradation on complex reasoning.

→ Check if a model fits your GPU | → Best GPU by VRAM tier