Evidence: benchmark results vendor-reported
Granite Switch 4.1 8B — VRAM, speed & local setup
Written by Jakub Rusinowski · Last updated
Granite 4.1 8B composed with two aLoRA adapters (decision and guardian-core) and an encoder: 8.8B parameters in one checkpoint, 128K context. Control tokens in the chat template pick the adapter per request. Revision 4.1.1 (27 October 2026) updated the decision adapter. Official GGUF; llama.cpp has supported the decision adapter since 16 October 2026.
Granite Switch 4.1 8B needs about 6 GB of VRAM at Q4_K_M — quantized weights plus framework overhead, before any KV cache. On Apple Silicon that figure comes out of unified memory.
VRAM and speed by quantization
Quoted against NVIDIA RTX 4090 (24 GB). Includes the KV cache at 8K context, so it reads higher than the headline figure.
| Quant | Memory | VRAM | Speed (est.) | Fit |
|---|---|---|---|---|
| Q2_K 2.63 bpw | 5.1 GB | ~143 tok/s | Fits | |
| Q3_K_M 3.41 bpw | 6 GB | ~123 tok/s | Fits | |
| Q4_K_M 4.83 bpw | 7.5 GB | ~98 tok/s | Fits | |
| Q5_K_M 5.67 bpw | 8.4 GB | ~88 tok/s | Fits | |
| Q6_K 6.56 bpw | 9.4 GB | ~79 tok/s | Fits | |
| Q8_0 8.50 bpw | 11.5 GB | ~64 tok/s | Fits | |
| F16 16.00 bpw | 19.8 GB | ~38 tok/s | Fits |
Black marker = usable memory on the NVIDIA RTX 4090 (24 GB). Estimates from the memory-bandwidth roofline on the methodology page. Granite Switch 4.1 8B VRAM calculator →
Get Granite Switch 4.1 8B running
The cheapest catalogued GPU that runs Granite Switch 4.1 8B is the Intel Arc B570 (10 GB).
or compare on Vast.ai from $0.35/hr (typical low · varies)
As an Amazon Associate we earn from qualifying purchases. Cloud GPU links are referral links — we may earn a commission at no extra cost to you.
How to run Granite Switch 4.1 8B
No Ollama tag. Use IBM's official GGUF with llama.cpp.
Specifications
Verified — Checked against the primary source — the model card or the vendor spec page — and corroborated by a second independent source.
- Parameters
- 8.8 Billion (base + adapters + encoder)
- Context window
- 131,072
- Architecture
- Dense Transformer (GQA) with aLoRA adapters
- Provider
- IBM
- Licence
- Apache-2.0
- Practical deployment
- Laptop
- Specified at
- Q4_K_M
- System RAM
- 16 GB
- Record updated
- 2026-11-03
Commercial use permitted. No usage restrictions beyond attribution.
Limits and caveats
- A composed checkpoint: the adapters are reached through control tokens, so a plain chat client uses only the base model.
- Kept out of general chat rankings; compare it with Granite 4.1 8B for plain chat.
See also Granite 4.1 8B
Quality and use cases
Scores as published by the model’s authors or an independent evaluator — quality, not throughput, and not measured by us.
Other IBM Granite Switch 4.1 sizes
Granite Switch 4.1 8B — frequently asked questions
How much VRAM does Granite Switch 4.1 8B need?
About 6 GB at Q4_K_M — quantized weights plus framework overhead, before any KV cache. The cache grows with context length and is added on top; the table above folds it in. Apple Silicon counts unified memory toward the same figure.
Does Granite Switch 4.1 8B run on an RTX 4090 (24 GB)?
Yes. Granite Switch 4.1 8B needs about 6 GB at Q4_K_M, inside a 24 GB card, at an estimated 98 tokens/sec.
How do I run Granite Switch 4.1 8B locally?
No Ollama tag. Use IBM's official GGUF with llama.cpp. Running the published tag would send your prompts to a hosted GPU rather than your own machine.
What other sizes does IBM Granite Switch 4.1 come in?
Granite Switch 4.1 8B (6 GB), Granite Switch 4.1 30B (19 GB). Every size shares the family's training and licence; the larger ones score higher and need proportionally more memory.