Best Local LLMs for RAG
作者： Jakub Rusinowski · 最后更新： 2026年9月6日
Answering from a private document set: the model reads retrieved passages and must stay faithful to them.
Top pick: Gemma 4 31B
Scores 95.2/100 for retrieval-augmented generation over your own documents. 31B parameters, needing about 19.5 GB at Q4_K_M, 250K context, Apache-2.0.
Ranked for retrieval-augmented generation over your own documents
| Model | Score | Params | Context | Licence | Quality index |
|---|---|---|---|---|---|
| 1. Gemma 4 31B | 95.2 | 31B | 250K | Apache-2.0 | — (estimated) |
| 2. Qwen 3.6 27B | 94.5 | 28B | 256K | Apache-2.0 | — (estimated) |
| 3. Qwen3.8 27B | 93.7 | 28B | 256K | Apache-2.0 | — (estimated) |
| 4. Qwen 3.7 35B-A3B | 93 | 35B | 256K | Apache-2.0 | — (estimated) |
| 5. Ternary Bonsai 27B | 90.2 | 27B | 256K | Apache 2.0 | — (estimated) |
| 6. DeepSeek V4-Flash | 90.2 | 284B | 977K | MIT | — (estimated) |
Frontier open-weight models (multi-GPU / private cloud)
These also score for retrieval-augmented generation over your own documents, but none runs on a single consumer GPU or workstation, so they are listed apart rather than ranked against the local picks above.
- DeepSeek V4.1 Flash — Multi-GPU server
- MiMo-V2.5-Pro 1T — Private cloud / datacenter cluster
- Inkling (NVFP4) — Private cloud / datacenter cluster
- DeepSeek V4-Pro — Private cloud / datacenter cluster
- Mistral Large 3 675B-A41B — Multi-GPU server
Best pick for your memory budget
The strongest model overall is rarely the right answer — what matters is the strongest model that fits the memory you have. These picks are re-ranked per tier, so each one uses its budget rather than simply being small.
| Memory | Typical hardware | Recommended models |
|---|---|---|
| 8 GB | RTX 4060, RTX 3070, base MacBook Air | Ternary Bonsai 27B (90.2) Ternary Bonsai 2 27B (86.1) GLM-4 9B (79.3) |
| 12 GB | RTX 3060 12 GB, RTX 5070 | Ternary Bonsai 27B (89.7) Ternary Bonsai 2 27B (85.7) Qwen 3 14B (84.4) |
| 16 GB | RTX 5080, RTX 4080, RX 9070 XT | Devstral Small 2 24B (88.4) Ternary Bonsai 27B (88.4) Qwen 3 14B (84.4) |
| 24 GB | RTX 4090, RTX 3090, RX 7900 XTX | Gemma 4 31B (97.5) Qwen 3.6 27B (96.7) Qwen3.8 27B (95.9) |
| 48 GB | RTX 6000 Ada, MacBook Pro M4 Max 48 GB | Gemma 4 31B (95.6) Qwen 3.6 27B (94.5) Qwen 3.7 35B-A3B (93.8) |
| 128 GB+ | Mac Studio, DGX Spark, multi-GPU | Llama 4 Scout 17B (93.6) Qwen3.8-Flash-Next (93) Gemma 4 31B (92.8) |
How this ranking works
Context is the binding constraint, so the floor is the highest of any text workload (32K) and full marks need 256K. Reasoning carries the capability weight because RAG failures are almost always synthesis failures, not retrieval failures. Licence importance is raised to 0.6 — RAG is overwhelmingly deployed on private or commercial corpora.
Worked example — Gemma 4 31B: capability 92.8 × 0.418, quality 91.7 × 0.246, context 96.9 × 0.16, license 100 × 0.066, accessibility 80 × 0.111 + 3 tag bonus (long-context).
Requirements applied: context floor 65,536 tokens (ideal 262,144), quality floor 55, licence weight 0.6, latency weight 0.5.
Running retrieval-augmented generation over your own documents locally
- KV cache, not weights, is what actually exhausts memory in RAG: at 128K context the cache can exceed the model itself. Quantizing the KV cache to q8_0 roughly halves it.
- Retrieval quality caps answer quality. A stronger model cannot rescue passages that never surfaced.
FAQ
What is the best local LLM for retrieval-augmented generation over your own documents?
Gemma 4 31B, scoring 95.2/100 against this workload's published requirements. 159 models qualified.
What hardware do I need for retrieval-augmented generation over your own documents?
A credible answer starts at 8 GB of memory. Larger budgets unlock materially stronger models — the table above lists the best pick at each tier.
How were these models ranked?
Context is the binding constraint, so the floor is the highest of any text workload (32K) and full marks need 256K. Reasoning carries the capability weight because RAG failures are almost always synthesis failures, not retrieval failures. Licence importance is raised to 0.6 — RAG is overwhelmingly deployed on private or commercial corpora.
Hardware for This Workload
- Best GPU for rag
- Best models for the NVIDIA GeForce RTX 4060 Ti 8GB
- Best models for the NVIDIA GeForce RTX 3080 Ti
- Best models for the NVIDIA GeForce RTX 4090 Laptop GPU
Related Workloads
- Best local LLMs for research
- Best local LLMs for privacy
- Best local LLMs for vision
- Best local LLMs for offline ai