How much VRAM does your context window need?
Pick a model, a context length and a KV cache type. You get the memory the cache takes, the weights, the total, and the exact setting to use in Ollama, llama.cpp, LM Studio, vLLM or MLX.
Advanced: flash attention, ubatch, sliding window
- Full-attention layers: 4.3 GB (32,768 cells after padding)
- llama.cpp’s compute buffer sits inside the overhead; at ubatch 512 it is typically 0.3–1 GB (an estimate, not from a source).
llama-server -m model.gguf -c 32768 -fa onWill it fit? Check your GPU in the analyzer — it owns the fit verdict.
Arithmetic from each model’s config.json, using llama.cpp’s allocation rules (commit f3fb2fc). Figures in GB are 10⁹ bytes, like the rest of this site.
KV cache at 8K, 32K and 128K
| Model | 8K f16 | 8K q8_0 | 32K f16 | 32K q8_0 | 128K f16 | 128K q8_0 |
|---|---|---|---|---|---|---|
| Llama 3.1 8B Instruct | 1.1 GB | 0.6 GB | 4.3 GB | 2.3 GB | 17.2 GB | 9.1 GB |
| Qwen3.8 27B | 0.7 GB | 0.4 GB | 2.3 GB | 1.3 GB | 8.7 GB | 4.7 GB |
| Gemma 4 31B | 1.9 GB | 1 GB | 3.9 GB | 2.1 GB | 12 GB | 6.4 GB |
How it works
The cache is sized layer by layer from each model’s config.json, following llama.cpp’s allocation rules. Sliding-window layers keep only their window, hybrid models keep a small fixed state, and quantised caches include their block overhead.
- Choose a model (only models with a published attention config are listed), or enter layers, KV heads and head size yourself.
- Set the context you want per request, the KV cache type and the number of parallel slots.
- Pick your runtime. It changes how slots multiply the cache and which setting to copy.
- Check the total against your GPU in the analyzer, which owns the fit verdict.
Sources
- llama.cpp source at f3fb2fc (llama-kv-cache.cpp, llama-kv-cache-iswa.cpp, llama-context.cpp) — allocation rules, padding, slots
- ggml block formats (ggml-common.h) — q8_0 = 34 bytes per 32 values, q4_0 = 18
- Ollama source at 36a6e99 (llama_server.go, sched.go, routes.go) — num_ctx × parallel, default context tiers
- Ollama context length docs
- llama.cpp PR 7412: KV cache quantisation perplexity and KL divergence
- llama.cpp issue 10560: a real q8_0 KV cache log (Mistral Large 2411) — reproduced exactly
- ik_llama.cpp issue 1607: Gemma 4 26B-A4B cache log — reproduced exactly
- Hugging Face: Llama 3.1 blog, KV cache table — its 8B and 70B rows match
Checked 2026-10-19.
Questions
How much VRAM does 128k context need?
It depends on the model’s attention layout far more than its size. With an f16 cache at 131,072 tokens: Llama 3.1 8B Instruct: 17.2 GB; Llama 3.3 70B Instruct: 42.9 GB; Mistral Small 3.2 24B: 21.5 GB; Qwen3.8 27B: 8.7 GB; Gemma 4 31B: 12 GB; GPT-OSS 120B: 4.9 GB. That is on top of the weights. A q8_0 cache needs a little over half of it.
How much VRAM does 64k context use?
About half of the 128k figure, because full-attention layers grow linearly with context while sliding-window and recurrent parts stay fixed. Llama 3.1 8B needs 8.6 GB of f16 cache at 64k and 17.2 GB at 128k.
Why does my model run out of memory at long context?
In order of how often it happens: the runtime picked a large context for you (llama-server loads the trained context unless --fit lowers it; Ollama defaults to 256k on machines with 48 GB or more of VRAM); parallel slots (OLLAMA_NUM_PARALLEL=4 makes four full caches); flash attention is off, so the attention scratch grows with context; --swa-full or a fork that allocates full sliding-window caches; and images, which add a vision projector and use context tokens.
Does quantising the KV cache hurt quality?
q8_0 is close to lossless: in the llama.cpp maintainers’ test, perplexity moved from 6.232 to 6.234 with q8_0 K and V (PR 7412, May 2024). q4_0 on the K cache is the risk — perplexity rose to 6.438 — while q4_0 on V alone stayed small. Use q8_0 when you need room: it takes Llama 3.1 8B at 128k from 17.2 GB to 9.1 GB. A quantised V cache needs flash attention.
What context does Ollama use by default?
It depends on your GPU memory: 4,096 tokens below about 24 GB of VRAM, 32,768 up to about 48 GB, and 262,144 above that, capped at the model’s trained context. Set your own with OLLAMA_CONTEXT_LENGTH, num_ctx in the API, or PARAMETER num_ctx in a Modelfile.
Do parallel slots multiply the KV cache?
In Ollama, yes: it starts llama.cpp with num_ctx × OLLAMA_NUM_PARALLEL cells, so four slots make four caches (Ollama forces one slot for Qwen3.5-family hybrids). In llama-server, -c is the total shared by all slots, so to give each of four slots 32k you pass -c 131072.
Why do other calculators give a bigger number?
Usually one of four reasons: they charge sliding-window layers for the full context (Gemma and gpt-oss come out several times too large), they apply the head count to MLA models like DeepSeek, they mix GB and GiB or 128,000 and 131,072 tokens, or they ignore Ollama’s parallel-slot multiplier.