Mistral Small 3.1 24B at Q4_K_M needs 16.4 GB once weights, KV cache at 8K context and framework overhead are counted. RTX 5080 16GB offers 16 GB, leaving you 0.4 GB short. You are 0.4 GB short in VRAM but have 25.6 GB of usable system RAM, so llama.cpp can hold the overflow layers in RAM instead of refusing to load. Expect 19.5–40.5 tok/s (estimated).
You are 0.4 GB short in VRAM but have 25.6 GB of usable system RAM, so llama.cpp can hold the overflow layers in RAM instead of refusing to load.
Your machine is not exactly this one. Run this for your exact setup — the form opens pre-filled with RTX 5080 16GB and Mistral Small 3.1 24B.
Not as it stands. Mistral Small 3.1 24B needs 16.4 GB at Q4_K_M and this machine has 16 GB available, a shortfall of 0.4 GB. You are 0.4 GB short in VRAM but have 25.6 GB of usable system RAM, so llama.cpp can hold the overflow layers in RAM instead of refusing to load.
Here, yes — up to a point. You are 0.4 GB short in VRAM, and 64 GB of system RAM gives llama.cpp somewhere to put the overflow layers without changing your graphics card. It works because the shortfall is small enough that the layers living in system RAM do not dominate each token.
New GPU prices are far above MSRP because the memory on the board costs several times what it did in 2025. Relief is not expected before 2027-Q4. The current estimate for relief is 2027-Q4. If you can run this some other way meanwhile, waiting is defensible; if you cannot, the part still does the job today.
Around 19.5–40.5 tok/s (estimated) on this setup at Q4_K_M. Decode speed on a local model is set by memory bandwidth rather than compute, so the figure moves with the card's bandwidth and the bytes read per token — not with its price.
Yes, and it is the figure people forget. The 16.4 GB above already includes the KV cache at 8K; that cache grows roughly linearly with context, so doubling the window adds real gigabytes rather than a rounding error. If you plan to work with long documents on RTX 5080 16GB, size for the context you will actually use, not the default.
Data behind this page last checked 2026-08-26.