Autor: Jakub Rusinowski · Ostatnia aktualizacja: 18 lipca 2024
Model library → Mistral Family → Mistral NeMo 12B
Collaboration with NVIDIA. Fits in 12GB VRAM with large context. Replaces Mistral 7B.
Mistral NeMo 12B needs about 8 GB of VRAM at Q4_K_M — quantized weights plus framework overhead, before any KV cache. On Apple Silicon that figure comes out of unified memory.
| Parameters | 12 Billion |
| Context window | 128,000 |
| Architecture | Dense Transformer |
| Provider | Mistral AI |
| Licence | Apache 2.0 |
| Specified at | Q4_K_M |
| System RAM | 16 GB |
| Record updated | 2024-07-18 |
Apache-2.0 — commercial use permitted. Commercial use permitted. No usage restrictions beyond attribution.
Modelled on a reference NVIDIA RTX 4090 (24 GB). Assumes an 8K-token context with an f16 KV cache. A longer window needs more; a quantized KV cache needs less. Speed figures are ESTIMATES from the memory-bandwidth roofline described on the methodology page, not benchmarks we ran — rows marked measured come from published or reader-submitted runs. VRAM here includes the KV cache, so it reads higher than the headline figure above, which does not.
| Quant | Bits/weight | Weights | VRAM needed | Est. speed | Fit on 24 GB |
|---|---|---|---|---|---|
| Q2_K | 2.63 | 3.9 GB | 6.1 GB | ~120 tok/s (est.) | Fits comfortably |
| Q3_K_M | 3.41 | 5.1 GB | 7.3 GB | ~101 tok/s (est.) | Fits comfortably |
| Q4_K_M | 4.83 | 7.2 GB | 9.4 GB | ~79 tok/s (est.) | Fits comfortably |
| Q5_K_M | 5.67 | 8.5 GB | 10.6 GB | ~70 tok/s (est.) | Fits comfortably |
| Q6_K | 6.56 | 9.8 GB | 12 GB | ~62 tok/s (est.) | Fits comfortably |
| Q8_0 | 8.50 | 12.8 GB | 14.9 GB | ~50 tok/s (est.) | Fits comfortably |
| F16 | 16.00 | 24 GB | 26.1 GB | ~4 tok/s (est.) | Offloads to system RAM (slow) |
Want to set your own context length and KV-cache quantization? Use the interactive VRAM calculator.
or compare on Vast.ai from $0.35/hr (typical low · varies)
As an Amazon Associate we earn from qualifying purchases. Cloud GPU links are referral links — we may earn a commission at no extra cost to you.
The cheapest catalogued GPU that runs Mistral NeMo 12B is the Intel Arc B570 (10 GB).
Install Ollama, then run:
ollama run mistral-nemoWeights on Hugging Face: mistralai/Mistral-Nemo-Instruct-2407.
Pick a quantization and open it in LM Studio, Ollama, or Jan, or download the raw .gguf file directly. Quant list and sizes resolved from Hugging Face.
| Quant | Size | Download (.gguf) |
|---|---|---|
| Q3_K_M | 5.12 GB (est.) | Mistral-Nemo-Instruct-2407-Q3_K_M.gguf |
| Q4_K_M | 7.25 GB (est.) | Mistral-Nemo-Instruct-2407-Q4_K_M.gguf |
| Q5_K_M | 8.51 GB (est.) | Mistral-Nemo-Instruct-2407-Q5_K_M.gguf |
| Q6_K | 9.84 GB (est.) | Mistral-Nemo-Instruct-2407-Q6_K.gguf |
| Q8_0 | 12.75 GB (est.) | Mistral-Nemo-Instruct-2407-Q8_0.gguf |
Download in LM Studio: lms get bartowski/Mistral-Nemo-Instruct-2407-GGUF
Want this model on your phone? You can run it on your desktop with LM Studio and chat from your iPhone or iPad over an encrypted link — see Run LM Studio Models on Your Phone (LM Link).
Best for: rag, chat.
← All Mistral Family models | VRAM calculator | Build a PC for this model | Check your own hardware