Best Small LLMs for Low-End Hardware — Running AI on 4 GB VRAM

The narrative around local AI tends to focus on high-end hardware: RTX 4090s, Apple M4 Max setups, multi-GPU rigs. But the majority of people interested in running AI locally have something more modest — an old gaming laptop with a GTX 1060, an office PC with integrated graphics, or a Raspberry Pi.

Good news: you can run genuinely useful AI on 4 GB of VRAM in 2026. The small model category has matured dramatically. Here are the best options and how to get the most from limited hardware.

The Hardware Reality: What 4 GB VRAM Gets You

4 GB of VRAM is the minimum for running quantized LLMs on GPU. At Q4_K_M quantization:

• 1B parameter model: ~0.8 GB — fits easily
• 3B parameter model: ~2 GB — fits with headroom for context
• 7–8B parameter model: ~4–5 GB — tight, may need Q3_K or Q2_K quantization or CPU offloading
• 13B+ model: won't fit in VRAM, falls back to CPU RAM (very slow)

GPUs in the 4 GB class: GTX 1650, GTX 1060 (6GB variant has more headroom), RTX 3050 4GB laptop, integrated Intel Arc.

Tip: If you have 8 GB system RAM available alongside 4 GB VRAM, Ollama and llama.cpp can split model layers between GPU VRAM and CPU RAM. This allows running 7B models with some layers on GPU (fast) and some on CPU (slower) — a meaningful speed improvement over CPU-only.

The 5 Best Models for 4 GB VRAM

1. Qwen 2.5 3B — The Best All-Rounder

Size at Q4_K_M: ~2 GB Tokens/sec on GTX 1650: ~18–22 t/s Best for: General chat, writing assistance, basic coding, summaries

Qwen 2.5 3B is the single best "I have limited hardware" recommendation for 2026. It punches dramatically above its weight class, with multilingual support (29 languages), solid coding ability, and reasoning quality that surprises people expecting something toy-like.

ollama pull qwen2.5:3b
ollama run qwen2.5:3b

Compared to the older Llama 3.2 3B (which was itself impressive), Qwen 2.5 3B shows noticeably better instruction following and code output. If you have one model on 4 GB hardware, make it this one.

2. Llama 3.2 3B — Meta's Compact Contender

Size at Q4_K_M: ~2 GB Tokens/sec on GTX 1650: ~20–25 t/s Best for: General tasks, US-English-primary workflows, API integration

Llama 3.2 3B is Meta's smallest widely-deployed model and benefits from the massive Llama ecosystem — fine-tunes, system prompts, community tools all work out of the box. It's slightly faster than Qwen 2.5 3B on English tasks but falls behind on multilingual and coding benchmarks.

ollama pull llama3.2:3b

If you're building something that needs to integrate with the broader Llama tooling ecosystem (fine-tuning pipelines, specialized system prompts developed for Llama), this is the better base.

3. Phi-4 Mini — Microsoft's Efficiency Experiment

Size at Q4_K_M: ~2.3 GB Tokens/sec on GTX 1650: ~16–20 t/s Best for: STEM questions, math, reasoning tasks

Microsoft's Phi series continues to push the "small but smart" envelope. Phi-4 Mini (3.8B parameters) achieves remarkable benchmark scores for its size, particularly on math and scientific reasoning. The architecture is trained specifically to pack more capability into fewer parameters.

ollama pull phi4-mini

Don't expect Phi-4 Mini to write great marketing copy — it feels "academic." But if you're asking it to help with calculations, explain technical concepts, or solve logic problems, it outperforms its size class significantly.

4. Gemma 3n E2B — Google's Mobile-Optimized Model

Size at Q4_K_M: ~1.3 GB Tokens/sec on GTX 1650: ~28–35 t/s Best for: Very constrained hardware, fast response needs, embedded systems

Gemma 3n uses Google's MatFormer architecture, which allows "nested" models that can run at different parameter counts from the same weights. The E2B variant (2 billion effective parameters) is extraordinarily efficient — it fits in under 1.5 GB of VRAM and runs faster than models twice its size.

ollama pull gemma3n:e2b

Quality is noticeably below Qwen 2.5 3B and Llama 3.2 3B on complex tasks, but for simple assistant tasks, quick lookups, and environments where speed matters more than depth (kiosk systems, embedded assistants, always-on background tools), Gemma 3n E2B is hard to beat.

5. Mistral 7B Q2_K — The Stretched Option

Size at Q2_K (aggressive quantization): ~3.1 GB Tokens/sec on GTX 1650: ~8–12 t/s (some CPU offloading) Best for: Users who need 7B quality and are willing to accept slower speed

Q2_K is aggressive quantization — it compresses the model heavily, losing some quality in exchange for size. Mistral 7B at Q2_K fits in 4 GB VRAM and delivers significantly better reasoning than any 3B model, at the cost of slower generation speed.

# In LM Studio: search "Mistral 7B" and select Q2_K quantization
# Or download directly from Hugging Face and use with llama.cpp

The quality hit from Q2_K is real — you'll notice more hallucinations and weaker reasoning compared to Q4_K_M. But if your task requires 7B-class capability and you're willing to wait a bit longer per response, this is the path.

CPU-Only: When You Have No GPU

If you have no discrete GPU (integrated graphics only, or no dedicated card), you're on CPU inference. It's slow — expect 2–5 t/s even on a modern Intel Core i7 or Ryzen 7.

Best models for CPU-only: 1. Qwen 2.5 1.5B — fast enough to be usable at 4–7 t/s, genuinely helpful 2. SmolLM2 1.7B — HuggingFace's efficiency-optimized tiny model, designed for edge deployment 3. Gemma 3n E2B — excellent CPU efficiency from the MatFormer architecture

ollama pull qwen2.5:1.5b
ollama pull smollm2:1.7b

At 4–7 t/s, conversations feel slow but workable. Long-form tasks (summarize this 50-page document) are painful. Short interactions (answer this question, fix this function) are usable.

Optimization Tips for Limited Hardware

1. Use Q4_K_M by default — not Q8_0 Q8_0 is higher quality but nearly twice the file size and VRAM usage. Q4_K_M offers 95%+ of the quality at half the VRAM. Only go to Q8_0 if you have plenty of headroom. 2. Reduce context length Every token in your context uses VRAM for the KV cache. In Ollama, reduce the context window:

ollama run qwen2.5:3b --context-size 2048

The default 4096+ context uses significantly more VRAM than 2048.

3. GPU offloading in Ollama Set OLLAMA_GPU_LAYERS=99 to push as many layers as possible to GPU, letting overflow go to CPU RAM. On 4 GB systems with 16 GB system RAM, you can run 7B models this way at acceptable speeds. 4. Keep the system tray clear Every running application uses VRAM (especially browsers with GPU acceleration enabled). Close Chrome/Edge before running large models on 4 GB systems.

What's Not Realistic on 4 GB

Be honest with yourself about expectations:

• Long document analysis (10,000+ token documents): context overflow will cause degradation or errors
• Complex multi-step reasoning chains: smaller models lose track of context in long chains
• High-quality creative writing: 3B models produce competent but rarely excellent prose
• Running multiple models simultaneously: won't work reliably on 4 GB

These limitations aren't dealbreakers — they're guardrails. Use 3B models for what they're genuinely good at: quick answers, code snippets, summaries of short documents, and a local AI assistant that costs $0/month.

Check which models fit your specific hardware → Hardware Analyzer Compare all model sizes and VRAM requirements → Model Library