Written by Jakub Rusinowski · Last updated August 15, 2026
Model library → IBM Granite 4.0 → Granite 4.0 Small-H 32B-A9B
The largest Granite 4.0 build — 32B total, 9B active, on a hybrid mamba-2 architecture (the '-h' suffix). Also published as `granite4:small-h`. The mamba-2 hybrid keeps memory growth flatter than a pure transformer at long context, which is the point for document-heavy enterprise RAG. Around 19 GB at the default Q4_K_M, so a single 24GB card. Twelve languages including English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch and Chinese.
Granite 4.0 Small-H 32B-A9B needs about 20 GB of VRAM at Q4_K_M — quantized weights plus framework overhead, before any KV cache. On Apple Silicon that figure comes out of unified memory.
| Parameters | 32 Billion (9B active) |
| Context window | 128,000 |
| Architecture | Hybrid Mamba-2 + Mixture-of-Experts |
| Provider | IBM |
| Licence | Apache 2.0 |
| Specified at | Q4_K_M |
| System RAM | 32 GB |
| Record updated | 2026-08-15 |
Apache-2.0 — commercial use permitted. Commercial use permitted. No usage restrictions beyond attribution.
Modelled on a reference NVIDIA RTX 4090 (24 GB), with no KV cache (this record has no published architecture). Speed figures are ESTIMATES from the memory-bandwidth roofline described on the methodology page, not benchmarks we ran — rows marked measured come from published or reader-submitted runs. VRAM here includes the KV cache, so it reads higher than the headline figure above, which does not.
| Quant | Weights | VRAM needed | Est. speed | Fit on 24 GB |
|---|---|---|---|---|
| Q2_K | 10.5 GB | 11.3 GB | ~135 tok/s (est.) | Fits comfortably |
| Q3_K_M | 13.6 GB | 14.4 GB | ~117 tok/s (est.) | Fits comfortably |
| Q4_K_M | 19.3 GB | 20.1 GB | ~94 tok/s (est.) | Fits comfortably |
| Q5_K_M | 22.7 GB | 23.5 GB | ~84 tok/s (est.) | Tight fit |
| Q6_K | 26.2 GB | 27.0 GB | ~11 tok/s (est.) | Offloads to system RAM (slow) |
| Q8_0 | 34.0 GB | 34.8 GB | ~9 tok/s (est.) | Offloads to system RAM (slow) |
| F16 | 64.0 GB | 64.8 GB | — | Won't fit |
Want the memory numbers alone, at every quantization level and your own context length? Use the Granite 4.0 Small-H 32B-A9B VRAM calculator.
or compare on Vast.ai from $0.35/hr (typical low · varies)
As an Amazon Associate we earn from qualifying purchases. Cloud GPU links are referral links — we may earn a commission at no extra cost to you.
The cheapest catalogued GPU that runs Granite 4.0 Small-H 32B-A9B is the AMD Radeon RX 7900 XT (20 GB).
Install Ollama, then run:
ollama run granite4:32b-a9b-hWeights on Hugging Face: ibm-granite/granite-4.0-h-small.
Best for: enterprise, rag, tool calling, multilingual, on premise.
← All IBM Granite 4.0 models | VRAM calculator | Check your own hardware