Mistral Medium 3.5 128B — VRAM, speed & local setup
Written by Jakub Rusinowski · Last updated
127.7B dense parameters over 88 layers, grouped-query attention with 8 KV heads, 262,144-token context, image input. Published in FP8 (267 GB on disk); Mistral recommends vLLM and points to Unsloth's GGUFs for llama.cpp, and Ollama carries a 128B tag. Its Modified MIT licence is not the attribution-only kind: a company whose consolidated monthly revenue exceeds $20M gets no rights under it.
Mistral Medium 3.5 128B needs about 78 GB of VRAM at Q4_K_M — quantized weights plus framework overhead, before any KV cache. On Apple Silicon that figure comes out of unified memory.
VRAM and speed by quantization
Quoted against NVIDIA RTX 4090 (24 GB). Includes the KV cache at 8K context, so it reads higher than the headline figure.
| Quant | Memory | VRAM | Speed (est.) | Fit |
|---|---|---|---|---|
| Q2_K 2.63 bpw | 45.7 GB | ~3 tok/s | Offload | |
| Q3_K_M 3.41 bpw | 58.2 GB | — | Too big | |
| Q4_K_M 4.83 bpw | 80.9 GB | — | Too big | |
| Q5_K_M 5.67 bpw | 94.3 GB | — | Too big | |
| Q6_K 6.56 bpw | 108.5 GB | — | Too big | |
| Q8_0 8.50 bpw | 139.4 GB | — | Too big | |
| F16 16.00 bpw | 259.2 GB | — | Too big |
Black marker = usable memory on the NVIDIA RTX 4090 (24 GB). Estimates from the memory-bandwidth roofline on the methodology page. Mistral Medium 3.5 128B VRAM calculator →
Get Mistral Medium 3.5 128B running
The cheapest catalogued GPU that runs Mistral Medium 3.5 128B is the NVIDIA DGX Spark (128 GB).
or compare on Vast.ai from $0.77/hr (typical low · varies)
As an Amazon Associate we earn from qualifying purchases. Cloud GPU links are referral links — we may earn a commission at no extra cost to you.
How to run Mistral Medium 3.5 128B
Install Ollama, then run:
ollama run mistral-medium-3.5:128bSpecifications
Verified — Checked against the primary source — the model card or the vendor spec page — and corroborated by a second independent source.
- Parameters
- 127.7 Billion
- Context window
- 262,144
- Architecture
- Dense Transformer (GQA) with vision encoder
- Provider
- Mistral AI
- Licence
- Modified MIT (revenue-gated)
- Availability
- Weights downloadable
- Practical deployment
- Workstation (large unified memory or pro GPU)
- Specified at
- Q4_K_M
- System RAM
- 128 GB
- Record updated
- 2026-10-15
Weights are downloadable and commercial use is permitted, subject to the licence’s acceptable-use terms.
No rights at all for a company whose consolidated monthly revenue exceeds $20M.
Limits and caveats
- Licence: no rights at all for a company above $20M consolidated monthly revenue.
- Dense 128B: every token reads all 128B parameters, so decode is far slower than a mixture-of-experts model of similar size; the estimate table on this page gives the figure for your hardware.
Quality and use cases
Scores as published by the model’s authors or an independent evaluator — quality, not throughput, and not measured by us.
Mistral Medium 3.5 128B — frequently asked questions
How much VRAM does Mistral Medium 3.5 128B need?
About 78 GB at Q4_K_M — quantized weights plus framework overhead, before any KV cache. The cache grows with context length and is added on top; the table above folds it in. Apple Silicon counts unified memory toward the same figure.
Does Mistral Medium 3.5 128B run on an RTX 4090 (24 GB)?
No. Mistral Medium 3.5 128B needs about 78 GB at Q4_K_M, more than a single RTX 4090's 24 GB. It needs a larger card, several GPUs, or Apple Silicon with enough unified memory — or it runs with part of the weights offloaded to system RAM, which is much slower.
How do I run Mistral Medium 3.5 128B locally?
Install Ollama and run `ollama run mistral-medium-3.5:128b`. That pulls the weights and starts a local OpenAI-compatible endpoint; after the download nothing leaves the machine.