SmolLM2 — local AI model by Hugging Face
Written by Jakub Rusinowski · Last updated
The smallest production-quality LLMs. Hugging Face's SmolLM2 models are designed to run on microcontrollers, phones, and browsers via WebAssembly. Despite tiny size, they show surprising intelligence thanks to careful data curation with the Smoltalk dataset.
Variants
The smallest SmolLM2 variant needs about 1 GB of VRAM at Q4_K_M — quantized weights plus framework overhead, before any KV cache.
| Model | VRAM at Q4 | VRAM | Context | Run it |
|---|---|---|---|---|
| SmolLM2 1.7B Instruct → 1.7B | ~1.8 GB | 8,192 | ollama run smollm2:1.7b | |
| SmolLM2 360M Instruct → 360M | ~1 GB | 8,192 | ollama run smollm2:360m |
Memory is quantized weights plus overhead at Q4_K_M, from the same engine as the GPU & VRAM checker.
How to run SmolLM2 locally
Install Ollama, then pull the tag.
ollama run smollm2:1.7bPick a size above for its own VRAM figure, speed estimate and install command.
Licence
Commercial use permitted. No usage restrictions beyond attribution.
Applies to: SmolLM2 1.7B Instruct, SmolLM2 360M InstructRecommended GPU
The cheapest catalogued GPU that runs SmolLM2 locally (min 1 GB VRAM) is the Intel Arc B570 (10 GB).
Can I run SmolLM2 on my GPU?
SmolLM2 — frequently asked questions
How much VRAM does SmolLM2 need?
SmolLM2 needs about 1 GB VRAM at Q4_K_M quantization for its smallest variant. Variants: SmolLM2 1.7B Instruct (2 GB, Q4_K_M); SmolLM2 360M Instruct (1 GB, Q4_K_M). On Apple Silicon, unified memory counts toward this requirement.
Can I run SmolLM2 on an RTX 4090 (24 GB)?
Yes — SmolLM2 runs on an RTX 4090 (24 GB) and other 24 GB cards such as the RTX 3090. Smaller variants also fit comfortably on 8–16 GB GPUs at Q4_K_M.
What quantization should I use for SmolLM2?
Q4_K_M is the best balance of quality and VRAM for SmolLM2 in most cases. Choose Q8_0 for near-lossless quality if you have spare VRAM, or smaller quants (Q3/Q2) only when memory is tight.
How do I run SmolLM2 with Ollama?
Install Ollama, then run: ollama run smollm2:1.7b. This downloads SmolLM2 and starts a local, OpenAI-compatible endpoint — no internet connection is needed after the initial download.