Mellum2.1 12B Thinking — VRAM, speed & local setup
Written by Jakub Rusinowski · Last updated
12.15B parameters, 2.5B active per token: 28 layers, 64 experts with 8 active, grouped-query attention with 4 KV heads, and a 1,024-token sliding window on three of every four layers, which keeps the KV cache small. 131,072-token context. JetBrains reports SWE-bench Verified 47.0 and LiveCodeBench v6 82.0 on its own harness. Served with vLLM on the card; JetBrains now also publishes official GGUF files (Q4_K_M 8.1 GB), although the card still calls GGUF "coming soon".
Mellum2.1 12B Thinking needs about 9 GB of VRAM at Q4_K_M — quantized weights plus framework overhead, before any KV cache. On Apple Silicon that figure comes out of unified memory.
VRAM and speed by quantization
Quoted against NVIDIA RTX 4090 (24 GB). Includes the KV cache at 8K context, so it reads higher than the headline figure.
| Quant | Memory | VRAM | Speed (est.) | Fit |
|---|---|---|---|---|
| Q2_K 2.63 bpw | 4.9 GB | ~293 tok/s | Fits | |
| Q3_K_M 3.41 bpw | 6.1 GB | ~268 tok/s | Fits | |
| Q4_K_M 4.83 bpw | 9 GB | ~231 tok/s | Fits | |
| Q5_K_M 5.67 bpw | 9.5 GB | ~214 tok/s | Fits | |
| Q6_K 6.56 bpw | 11.8 GB | ~199 tok/s | Fits | |
| Q8_0 8.50 bpw | 13.8 GB | ~171 tok/s | Fits | |
| F16 16.00 bpw | 25.2 GB | ~17 tok/s | Offload |
Black marker = usable memory on the NVIDIA RTX 4090 (24 GB). Estimates from the memory-bandwidth roofline on the methodology page. Mellum2.1 12B Thinking VRAM calculator →
Get Mellum2.1 12B Thinking running
The cheapest catalogued GPU that runs Mellum2.1 12B Thinking is the Intel Arc B570 (10 GB).
or compare on Vast.ai from $0.35/hr (typical low · varies)
As an Amazon Associate we earn from qualifying purchases. Cloud GPU links are referral links — we may earn a commission at no extra cost to you.
How to run Mellum2.1 12B Thinking
No Ollama library entry yet. JetBrains' official GGUF files are the local route; the card's own serving recipe is vLLM.
Specifications
Verified — Checked against the primary source — the model card or the vendor spec page — and corroborated by a second independent source. Still unconfirmed: releaseDate (Hugging Face repository date; trackers also give 8 October).
- Parameters
- 12.15 Billion (2.5B active)
- Context window
- 131,072
- Architecture
- Mixture-of-Experts (64 experts, 8 active), GQA with sliding window on 3 of 4 layers
- Provider
- JetBrains
- Licence
- Apache 2.0
- Availability
- Weights downloadable
- Practical deployment
- Laptop
- Specified at
- Q4_K_M
- System RAM
- 16 GB
- Record updated
- 2026-10-15
Commercial use permitted. No usage restrictions beyond attribution.
Limits and caveats
- Benchmark figures are JetBrains' own, run on one agent harness (Pi v0.73.1).
- Mainline llama.cpp support for the 'mellum' architecture could not be confirmed from source; treat the official GGUF as experimental until it is.
Quality and use cases
Scores as published by the model’s authors or an independent evaluator — quality, not throughput, and not measured by us.
| Benchmark | Score | Provenance |
|---|---|---|
| SWE-bench Verified | 47 % | vendor-claimed · JetBrains model card |
| LiveCodeBench v6 | 82 % | vendor-claimed · JetBrains model card |
Mellum2.1 12B Thinking — frequently asked questions
How much VRAM does Mellum2.1 12B Thinking need?
About 9 GB at Q4_K_M — quantized weights plus framework overhead, before any KV cache. The cache grows with context length and is added on top; the table above folds it in. Apple Silicon counts unified memory toward the same figure.
Does Mellum2.1 12B Thinking run on an RTX 4090 (24 GB)?
Yes. Mellum2.1 12B Thinking needs about 9 GB at Q4_K_M, inside a 24 GB card, at an estimated 231 tokens/sec.
How do I run Mellum2.1 12B Thinking locally?
No Ollama library entry yet. JetBrains' official GGUF files are the local route; the card's own serving recipe is vLLM. Running the published tag would send your prompts to a hosted GPU rather than your own machine.