Out of memory at long context (the KV cache, not the weights)
It loads fine, then dies once the conversation gets long
错误信息
llama_new_context_with_model: failed to allocate KV cache
| 如果你看到 | 原因是 | 前往 |
|---|---|---|
| It fails only on long inputs, never on short prompts | The cache at that length does not fit beside the weights | Fix 2: cap the context length |
| The load log fails right after the weights, mentioning the KV cache | The context you set is reserved in full at load time | Fix 2, then Fix 3 |
ollama ps shows a CONTEXT far larger than you use | A default or server-wide setting is reserving the cache | Fix 2: set num_ctx or OLLAMA_CONTEXT_LENGTH |
| You need the long context and cannot shorten it | Weights and cache together exceed the card | Fix 3: quantize the KV cache, or Fix 4 |
何时出现
The model loads without complaint and answers short prompts — then falls over once you feed it a long document or a long chat history. The crash comes at *generation* time on big inputs, not at load time, which is what makes it confusing: the weights clearly fit, yet it still runs out of memory.
- Pasting a long document or a large code file into a model that handled short questions fine.
- A chat that has grown over an afternoon until the history alone fills most of the window.
- Raising the context setting to the model's advertised maximum, such as 128K, without changing anything else.
- Serving several requests in parallel, where each parallel slot gets its own share of cache.
- Moving to a bigger GPU with a recent Ollama, which picks a longer default context on cards with 24 GB or more.
实际原因
Two different things draw on your VRAM: the model weights (fixed) and the KV cache (grows with context length). The KV cache stores keys/values for every token in the window, so memory use climbs roughly linearly with how many tokens you're holding. A model that fits comfortably at 4K context can need gigabytes more at 32K or 128K — and that extra is what tips a card that was previously "just fitting" into OOM. This is the classic Tight fit: fine on paper, no headroom for context. The size is easy to estimate. For every token, the cache holds a key and a value vector for each layer, so it scales with layers × KV heads × head size × context length × bytes per element. For Llama 3.1 8B at FP16 that is 128 KB per token: about 1 GB at 8K context and 4 GB at 32K. Runtimes also differ in when this bites. llama.cpp, Ollama and LM Studio reserve the whole cache when the model loads, so a large context setting fails at load time; Hugging Face transformers grows the cache as tokens arrive, so it fails partway through a long input.
如何修复
- 1
Leave VRAM headroom — size weights AND context together最常见的修复方法
The mistake is budgeting only for the weights. You need room for the weights plus the KV cache at the context length you actually use. The reliable way to avoid this is to pick a model/quant that leaves headroom on your card for the context you need — a "Tight" fit with no room for KV cache is exactly what OOMs on long inputs. Check what fits *with* your target context, not just the bare weights.
生效了吗？The analyzer's total for your model, quantization and context length sits below your VRAM with a margin, and the model loads at that context without the error.检查哪些模型适配你的硬件check what fits with your target context length, not just the weights打开显存检查器 → - 2
Cap the context length to what you need
If you don't need a 128K window, don't allocate one. Setting the context to the size of your actual inputs shrinks the KV cache directly. This is the single most effective knob. In Ollama it is a server setting or a per-request
num_ctx, and recent versions choose the default from your VRAM, so check what is actually allocated rather than assuming 4K.# llama.cpp: cap the context ./llama-cli -m model.gguf -c 8192 # Ollama, inside an `ollama run` session: >>> /set parameter num_ctx 8192 # Ollama, for every model: set it on the SERVER and restart it. # Prefixing `ollama run` does nothing — the client does not read it. OLLAMA_CONTEXT_LENGTH=8192 ollama serve生效了吗？ollama psshows the CONTEXT you set, or llama.cpp's load log reports ann_ctxequal to your value, and memory use falls accordingly. - 3
Quantize the KV cache
llama.cpp can store the KV cache at lower precision (e.g. q8_0 instead of f16), roughly halving its memory for a small quality cost. Useful when you genuinely need a long context on a tight card.
./llama-cli -m model.gguf -c 32768 \ --cache-type-k q8_0 --cache-type-v q8_0生效了吗？The load log reports the cache types you asked for and a smaller KV buffer. If llama.cpp refuses a quantized V cache, turn flash attention on with-fa on; older builds take-faon its own. - 4
Use a smaller-weight model to free room for context
If long context is non-negotiable, a smaller or more aggressively quantized model frees the VRAM the KV cache needs. Trading a little weight precision for a lot of usable context is often the right call for document work.
生效了吗？The smaller model loads at your full target context, with VRAM to spare innvidia-smior Activity Monitor.
如果以上都没用
If the context is capped and the error remains, the weights alone are too close to the card's limit, which the out-of-memory page covers. On Apple Silicon the same overflow shows up as swapping rather than an error, and on Windows as a sudden slowdown.
- CUDA out of memory — why it happens and how to fix it
- Your Mac is swapping to disk instead of running the model
- Windows shared GPU memory makes your model crawl
- The exact error, and the load-log lines showing n_ctx and the KV buffer size
- Model, quantization and the context length you set
- GPU and VRAM, or your Mac's unified memory
- Runtime and version:
ollama -v, or the llama.cpp build number
相关内容
常见问题
Why does my model OOM only on long prompts?
Because the KV cache grows with context length. Short prompts use little of it; long documents or chats need much more, and that extra memory is what pushes a tightly-fitting model over its VRAM limit.
How much VRAM does context use?
It depends on the model, but it scales roughly linearly with token count and can reach several gigabytes at very long contexts. That is why you should budget for weights plus context together, not weights alone.
Does quantizing the KV cache hurt quality?
Q8 KV cache is close to lossless in practice and roughly halves cache memory. More aggressive cache quantization can affect long-context coherence, so q8_0 is the usual sweet spot.
Can I quantize the KV cache in Ollama?
Yes. Set OLLAMA_KV_CACHE_TYPE=q8_0 on the Ollama server; q4_0 saves more at a larger quality cost. The setting applies to every model, and a quantized cache needs flash attention, which Ollama enables automatically where the hardware supports it and which OLLAMA_FLASH_ATTENTION=1 forces on.
Why does Ollama use a longer context on my new GPU?
Recent Ollama versions pick the default context from available VRAM: 4K tokens below 24 GB, 32K from 24 to 48 GB, and 256K at 48 GB or more, according to its documentation. A larger default reserves a larger cache. Set OLLAMA_CONTEXT_LENGTH, or num_ctx per request, if you want a fixed size.
Is a model's advertised context length usable on my hardware?
Only if the cache fits. A model's maximum context is what it was trained to handle, not what your card can hold: a 128K window on Llama 3.1 8B needs about 16 GB of FP16 cache on top of the weights. Choose the context you actually use, not the maximum the model card lists.