You don't need a $2000 GPU to run AI. Small Language Models (SLMs) are efficient enough for edge devices.
Recommended Models
- Phi-3.5 Mini (3.8B): Microsoft's marvel. Runs on almost anything.
- Qwen 2.5 1.5B/3B: Incredible coding performance for the size.
- Gemma 2 2B: Google's lightweight model.
Raspberry Pi 5 Setup
- Install a 64-bit OS (Ubuntu or Raspberry Pi OS).
- Install Ollama:
curl -fsSL https://ollama.com/install.sh | sh - Run a tiny model:
ollama run qwen2.5:1.5b - Expectation: 3-5 tokens per second. Usable for simple commands.
Laptop Optimization (Battery Life)
Running LLMs drains battery fast.
- Use GGUF format: It is optimized for CPU efficiency.
- Limit Threads: In Ollama, you can set
OMP_NUM_THREADS=4to prevent it from using all cores, saving power.
To size a specific model, use the VRAM calculator: the memory it needs at every quantization and context length.