Written by Jakub Rusinowski · Last updated July 10, 2026
Last Updated: May 2026 — MoE (Mixture of Experts) architecture is behind some of the most exciting models of 2026: Llama 4, DeepSeek V3, Qwen 3 MoE, and Gemma 4. Understanding MoE helps you choose the
Last Updated: May 2026 — MoE (Mixture of Experts) architecture is behind some of the most exciting models of 2026: Llama 4, DeepSeek V3, Qwen 3 MoE, and Gemma 4. Understanding MoE helps you choose the right model and manage your VRAM more effectively.
In a traditional "dense" LLM, every parameter is active for every token. In a Mixture of Experts model, the architecture is split into many specialized sub-networks (the "experts"), and only a small fraction are activated per token:
Dense model (e.g., Llama 3.1 8B):
MoE model (e.g., Qwen 3.5 35B-A3B):
The key insight: you store more knowledge (more total parameters) but only pay the compute cost of a smaller model.
| Property | Dense 7B | MoE 35B-A3B |
|---|---|---|
| Total parameters | 7B | 35B |
| Active parameters/token | 7B | ~3B |
| VRAM needed (Q4) | ~5 GB | ~20 GB |
| Inference speed (tokens/sec) | 60 t/s (RTX 4090) | 25–35 t/s (RTX 4090) |
| Knowledge/capability | 7B level | ~25–30B equivalent |
| Reasoning quality | Good | Significantly better |
You get the knowledge of a 35B model at roughly the inference cost of a 7B model — at the expense of more VRAM.
ollama run llama4:scoutollama run llama4:maverickThe original widely-adopted MoE model:
MoE models have an unusual VRAM profile:
Dense Llama 3.1 70B Q4:
VRAM: 40 GB (must fit all weights)
Compute/token: 70B ops
Typical speed: 5-8 t/s on RTX 4090
MoE Llama 4 Scout Q4:
VRAM: 10.5 GB (all experts still loaded)
Compute/token: 17B ops
Typical speed: 25-35 t/s on RTX 4090
MoE Llama 4 Maverick Q4:
VRAM: 24 GB (all experts still loaded)
Compute/token: 17B ops
Typical speed: 15-25 t/s on RTX 4090Critical insight: MoE reduces compute (speed) but not VRAM — all experts must be loaded into memory even if only 2–4 are used per token. A 685B MoE model still needs ~370GB VRAM, even though only 37B parameters run per token.
Choose MoE when:
Choose Dense when:
Ollama handles MoE models automatically. Some tips:
# Check available VRAM before loading large MoE
ollama list
# For Llama 4 Scout (10.5GB) on an 8GB card - it will CPU offload
# Force full GPU offload if you have enough VRAM
OLLAMA_NUM_GPU=99 ollama run llama4:scout
# For Qwen 3.5 35B-A3B on RTX 4090 24GB:
ollama run qwen3.5:35b-a3b-q4_K_MMoE models often have very large context windows (because the architecture scales well):
| Model | Context Window |
|---|---|
| Llama 4 Scout | 128k tokens |
| Llama 4 Maverick | 128k tokens |
| DeepSeek V3.2 | 128k tokens |
| Qwen 3.5 35B-A3B | 32k tokens |
| Gemma 4 26B-A4B | 128k tokens |
Large context + MoE efficiency = excellent for RAG, document analysis, and long-form tasks.
Does Ollama support MoE models automatically? Yes — Ollama handles MoE models transparently via llama.cpp. You don't need any special configuration.
Is MoE better than dense for my use case? For most interactive chat tasks, a dense 8B model is faster. For complex reasoning where you can wait 2–3 seconds per response, an MoE model with the same VRAM budget will give better answers.
Can I fine-tune MoE models? Technically yes with frameworks like LLaMA-Factory, but it's more complex. LoRA fine-tuning works but requires careful expert routing. Most users fine-tune dense models instead.
Why does a 685B model need 370GB VRAM if only 37B are active? All expert weight matrices must be in VRAM for the router to select from them. You can't know in advance which experts will be selected, so all must be loaded. This is the fundamental MoE memory constraint.