Choosing a local Large Language Model (LLM) is like choosing a car. You need to balance performance (speed), capacity (parameters), and fuel efficiency (VRAM). This guide breaks down exactly how to pick the perfect model for your setup.
1. The Golden Rule: VRAM is King
Unlike cloud models, local LLMs live entirely in your Video RAM (VRAM). If a model is larger than your VRAM, it spills over into system RAM (DDR4/DDR5), which is 10x-50x slower.
Quick Rule of Thumb:
- 8GB VRAM: Run 7B or 8B models (quantized to Q4_K_M).
- 12GB VRAM: Run 12B or 14B models (e.g., Mistral NeMo, Qwen 14B).
- 24GB VRAM: Run 27B-35B models (Gemma 2 27B, Command R) OR 70B models heavily quantized (Q2_K).
- Mac (Unified Memory): Apple Silicon uses Unified Memory. If you have 16GB RAM, you can effectively use ~11GB for models.
2. Parameter Size Explained
The number in the model name (e.g., Llama 3 8B) refers to the number of parameters (billions).
- < 4B (Small): Great for phones or Raspberry Pi. Fast but less smart. (Example: Phi-3.5)
- 7B - 9B (Medium): The sweet spot for most users. Good at general chat and simple coding. (Example: Llama 3.1 8B, Gemma 2 9B)
- 12B - 35B (Large): High intelligence, better reasoning. Requires 12GB+ VRAM. (Example: Qwen 2.5 14B, Gemma 2 27B)
- 70B+ (Massive): Research-grade intelligence. Slow on consumer hardware. (Example: Llama 3.1 70B)
3. Quantization: The Magic Shrink Ray
Models are trained in 16-bit precision (FP16). This makes them huge. Quantization reduces this precision to 4-bit (Q4) or even lower, shrinking the file size with minimal quality loss.
- Q4_K_M: The gold standard. Good balance of size vs smarts.
- Q8_0: Almost lossless, but big.
- Q2_K: Very compressed, noticeable 'brain damage'.
4. Use Case Recommendations
- For Coding: Choose Qwen 2.5 or DeepSeek Coder. They are fine-tuned on code repositories.
- For Roleplay: Choose Mistral or specialized 'finetunes' like Hermes.
- For RAG (Chat with Docs): Choose models with large context windows (128k) like Llama 3.1 or Mistral NeMo.
- For classification, routing or yes/no gates: Use a decision model such as Nimble or Tev1 rather than a chat model; it returns a probability for every option, and open alternatives beyond Ollama cover the larger ones. Decision models explained shows how they differ from chat models.
5. Next Steps
Go to the Hardware Check tab on this site to automatically find the best model for your rig.