Mistral NeMo 12B — VRAM, speed & local setup
Written by Jakub Rusinowski · Last updated
Collaboration with NVIDIA. Fits in 12GB VRAM with large context. Replaces Mistral 7B.
Mistral NeMo 12B needs about 8 GB of VRAM at Q4_K_M — quantized weights plus framework overhead, before any KV cache. On Apple Silicon that figure comes out of unified memory.
VRAM and speed by quantization
Quoted against NVIDIA RTX 4090 (24 GB). Includes the KV cache at 8K context, so it reads higher than the headline figure.
| Quant | Memory | VRAM | Speed (est.) | Fit |
|---|---|---|---|---|
| Q2_K 2.63 bpw | 6.1 GB | ~120 tok/s | Fits | |
| Q3_K_M 3.41 bpw | 7.3 GB | ~101 tok/s | Fits | |
| Q4_K_M 4.83 bpw | 9.4 GB | ~79 tok/s | Fits | |
| Q5_K_M 5.67 bpw | 10.6 GB | ~70 tok/s | Fits | |
| Q6_K 6.56 bpw | 12 GB | ~62 tok/s | Fits | |
| Q8_0 8.50 bpw | 14.9 GB | ~50 tok/s | Fits | |
| F16 16.00 bpw | 26.1 GB | ~4 tok/s | Offload |
Black marker = usable memory on the NVIDIA RTX 4090 (24 GB). Estimates from the memory-bandwidth roofline on the methodology page. Mistral NeMo 12B VRAM calculator →
Get Mistral NeMo 12B running
The cheapest catalogued GPU that runs Mistral NeMo 12B is the Intel Arc B570 (10 GB).
or compare on Vast.ai from $0.35/hr (typical low · varies)
As an Amazon Associate we earn from qualifying purchases. Cloud GPU links are referral links — we may earn a commission at no extra cost to you.
How to run Mistral NeMo 12B
Install Ollama, then run:
ollama run mistral-nemoDownload Mistral NeMo 12B — GGUF Quantizations
Pick a quantization and open it in LM Studio, Ollama, or Jan, or download the raw .gguf file directly. Quant list and sizes resolved from Hugging Face.
Mistral NeMo 12B — GGUF quants · bartowski/Mistral-Nemo-Instruct-2407-GGUF
| Quant | Size | Download (.gguf) |
|---|---|---|
| Q3_K_M | 5.12 GB (est.) | Mistral-Nemo-Instruct-2407-Q3_K_M.gguf |
| Q4_K_M | 7.25 GB (est.) | Mistral-Nemo-Instruct-2407-Q4_K_M.gguf |
| Q5_K_M | 8.51 GB (est.) | Mistral-Nemo-Instruct-2407-Q5_K_M.gguf |
| Q6_K | 9.84 GB (est.) | Mistral-Nemo-Instruct-2407-Q6_K.gguf |
| Q8_0 | 12.75 GB (est.) | Mistral-Nemo-Instruct-2407-Q8_0.gguf |
Download in LM Studio: lms get bartowski/Mistral-Nemo-Instruct-2407-GGUF
Want this model on your phone? You can run it on your desktop with LM Studio and chat from your iPhone or iPad over an encrypted link — see Run LM Studio Models on Your Phone (LM Link).
Specifications
Curated — A hand-written entry from before this catalogue recorded its sources. The figures are long-standing but their provenance is not on file.
- Parameters
- 12 Billion
- Context window
- 128,000
- Architecture
- Dense Transformer
- Provider
- Mistral AI
- Licence
- Apache 2.0
- Practical deployment
- Laptop
- Specified at
- Q4_K_M
- System RAM
- 16 GB
- Record updated
- 2024-07-18
Commercial use permitted. No usage restrictions beyond attribution.
Quality and use cases
Scores as published by the model’s authors or an independent evaluator — quality, not throughput, and not measured by us.
Can I run Mistral NeMo 12B on my GPU?
- Mistral NeMo 12B on AMD Radeon RX 7800 XT
- Mistral NeMo 12B on AMD Radeon RX 7900 XT
- Mistral NeMo 12B on AMD Radeon RX 7900 XTX
- Mistral NeMo 12B on AMD Radeon RX 9060 XT 16GB
- Mistral NeMo 12B on AMD Radeon RX 9060 XT 8GB
- Mistral NeMo 12B on AMD Radeon RX 9070
- Mistral NeMo 12B on AMD Radeon RX 9070 XT
- Mistral NeMo 12B on Apple M1
- Mistral NeMo 12B on Apple M1 Pro
- Mistral NeMo 12B on Apple M2
- Mistral NeMo 12B on Apple M2 Pro
- Mistral NeMo 12B on Apple M3
- Mistral NeMo 12B on Apple M3 Pro
- Mistral NeMo 12B on Apple M4
- Mistral NeMo 12B on Apple M4 Pro
- Mistral NeMo 12B on Apple M5
- Mistral NeMo 12B on Intel Arc B570
- Mistral NeMo 12B on Intel Arc B580
- Mistral NeMo 12B on NVIDIA GeForce RTX 3060 (12GB)
- Mistral NeMo 12B on NVIDIA GeForce RTX 3070
- Mistral NeMo 12B on NVIDIA GeForce RTX 3070 Ti
- Mistral NeMo 12B on NVIDIA GeForce RTX 3080 (10GB)
- Mistral NeMo 12B on NVIDIA GeForce RTX 3090
- Mistral NeMo 12B on NVIDIA GeForce RTX 4060
- Mistral NeMo 12B on NVIDIA GeForce RTX 4060 Ti 16GB
- Mistral NeMo 12B on NVIDIA GeForce RTX 4070
- Mistral NeMo 12B on NVIDIA GeForce RTX 4070 Super
- Mistral NeMo 12B on NVIDIA GeForce RTX 4070 Ti
- Mistral NeMo 12B on NVIDIA GeForce RTX 4070 Ti Super
- Mistral NeMo 12B on NVIDIA GeForce RTX 4080
- Mistral NeMo 12B on NVIDIA GeForce RTX 4080 Super
- Mistral NeMo 12B on NVIDIA GeForce RTX 4090
- Mistral NeMo 12B on NVIDIA GeForce RTX 5060
- Mistral NeMo 12B on NVIDIA GeForce RTX 5060 Ti 16GB
- Mistral NeMo 12B on NVIDIA GeForce RTX 5060 Ti 8GB
- Mistral NeMo 12B on NVIDIA GeForce RTX 5070
- Mistral NeMo 12B on NVIDIA GeForce RTX 5070 Ti
- Mistral NeMo 12B on NVIDIA GeForce RTX 5080
- Mistral NeMo 12B on NVIDIA GeForce RTX 5090
Other Mistral Family sizes
Mistral NeMo 12B — frequently asked questions
How much VRAM does Mistral NeMo 12B need?
About 8 GB at Q4_K_M — quantized weights plus framework overhead, before any KV cache. The cache grows with context length and is added on top; the table above folds it in. Apple Silicon counts unified memory toward the same figure.
Does Mistral NeMo 12B run on an RTX 4090 (24 GB)?
Yes. Mistral NeMo 12B needs about 8 GB at Q4_K_M, inside a 24 GB card, at an estimated 79 tokens/sec.
How do I run Mistral NeMo 12B locally?
Install Ollama and run `ollama run mistral-nemo`. That pulls the weights and starts a local OpenAI-compatible endpoint; after the download nothing leaves the machine.
What other sizes does Mistral Family come in?
Mistral Small 3 (24B) (15 GB), Mistral NeMo 12B (8 GB). Every size shares the family's training and licence; the larger ones score higher and need proportionally more memory.