How to Choose the Right Local LLM for Your Hardware

Written by Jakub Rusinowski · Last updated July 10, 2026

Choosing a local Large Language Model (LLM) is like choosing a car. You need to balance performance (speed), capacity (parameters), and fuel efficiency (VRAM). This guide breaks down exactly how to pi

In This Guide

• 1. The Golden Rule: VRAM is King
• 2. Parameter Size Explained
• 3. Quantization: The Magic Shrink Ray
• 4. Use Case Recommendations
• 5. Next Steps

Choosing a local Large Language Model (LLM) is like choosing a car. You need to balance performance (speed), capacity (parameters), and fuel efficiency (VRAM). This guide breaks down exactly how to pick the perfect model for your setup.

1. The Golden Rule: VRAM is King

Unlike cloud models, local LLMs live entirely in your Video RAM (VRAM). If a model is larger than your VRAM, it spills over into system RAM (DDR4/DDR5), which is 10x-50x slower.

Quick Rule of Thumb:

• 8GB VRAM: Run 7B or 8B models (quantized to Q4_K_M).
• 12GB VRAM: Run 12B or 14B models (e.g., Mistral NeMo, Qwen 14B).
• 24GB VRAM: Run 27B-35B models (Gemma 2 27B, Command R) OR 70B models heavily quantized (Q2_K).
• Mac (Unified Memory): Apple Silicon uses Unified Memory. If you have 16GB RAM, you can effectively use ~11GB for models.

2. Parameter Size Explained

The number in the model name (e.g., Llama 3 8B) refers to the number of parameters (billions).

• < 4B (Small): Great for phones or Raspberry Pi. Fast but less smart. (Example: Phi-3.5)
• 7B - 9B (Medium): The sweet spot for most users. Good at general chat and simple coding. (Example: Llama 3.1 8B, Gemma 2 9B)
• 12B - 35B (Large): High intelligence, better reasoning. Requires 12GB+ VRAM. (Example: Qwen 2.5 14B, Gemma 2 27B)
• 70B+ (Massive): Research-grade intelligence. Slow on consumer hardware. (Example: Llama 3.1 70B)

3. Quantization: The Magic Shrink Ray

Models are trained in 16-bit precision (FP16). This makes them huge. Quantization reduces this precision to 4-bit (Q4) or even lower, shrinking the file size with minimal quality loss.

• Q4_K_M: The gold standard. Good balance of size vs smarts.
• Q8_0: Almost lossless, but big.
• Q2_K: Very compressed, noticeable 'brain damage'.

4. Use Case Recommendations

• For Coding: Choose Qwen 2.5 or DeepSeek Coder. They are fine-tuned on code repositories.
• For Roleplay: Choose Mistral or specialized 'finetunes' like Hermes.
• For RAG (Chat with Docs): Choose models with large context windows (128k) like Llama 3.1 or Mistral NeMo.

5. Next Steps

Go to the Hardware Check tab on this site to automatically find the best model for your rig.

← All Guides | Check GPU Compatibility