Written by Jakub Rusinowski · Last updated July 10, 2026
Last Updated: May 2026 — The latest generation of local LLMs can see images, process documents, and even understand audio — not just generate text. This guide explains multimodal AI, which models supp
Last Updated: May 2026 — The latest generation of local LLMs can see images, process documents, and even understand audio — not just generate text. This guide explains multimodal AI, which models support it, and how to run vision models locally.
A multimodal model accepts multiple types of input: text, images, PDFs, audio, or video. The output is still typically text (descriptions, answers, analysis), though some models can also generate images.
Why it matters: You can now:
| Modality | Description | Local Support |
|---|---|---|
| Text → Text | Classic LLM inference | ✅ Universal |
| Image → Text | Describe, analyze, OCR images | ✅ Many models |
| PDF → Text | Extract and reason over document content | ✅ Via image chunking |
| Audio → Text | Transcription + understanding | ⚠️ Limited (Whisper + LLM pipeline) |
| Video → Text | Frame-by-frame analysis | ⚠️ Experimental |
| Text → Image | Generate images from prompts | ✅ Separate models (Stable Diffusion) |
Meta's vision model integrates seamlessly with Ollama:
# Pull and run Llama 3.2 Vision
ollama run llama3.2-vision:11b
# Then send an image in your chat
# (Works in Open WebUI with drag-and-drop)Google's Gemma 3 family is natively multimodal across all sizes:
ollama run gemma3:12b # Vision enabled by default
ollama run gemma3:27b # Best quality for complex imagesAlibaba's Qwen VL series excels at document and chart understanding:
ollama run qwen2.5-vl:7bVision models use more VRAM than text-only models of the same size because they include a vision encoder (typically CLIP or SigLIP):
| Model | VRAM (Q4_K_M) | VRAM (Q8) | Notes |
|---|---|---|---|
| Gemma 3 4B Vision | ~4 GB | ~7 GB | Fastest, entry-level |
| Llama 3.2 Vision 11B | ~7.5 GB | ~12 GB | Good all-rounder |
| Gemma 3 12B Vision | ~8.5 GB | ~14 GB | Best mid-range |
| Qwen 2.5-VL 7B | ~5.5 GB | ~8 GB | Best for documents |
| Gemma 3 27B Vision | ~17 GB | ~28 GB | Near GPT-4V quality |
| Qwen 2.5-VL 72B | ~42 GB | — | Multi-GPU required |
Open WebUI provides the easiest interface for vision tasks:
# Start Open WebUI with Ollama backend
docker run -d -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
--name open-webui \
ghcr.io/open-webui/open-webui:main
# Access at http://localhost:3000
# Drag and drop images directly into the chatFor programmatic use via the API:
import ollama
import base64
# Load and encode image
with open("screenshot.png", "rb") as f:
image_data = base64.b64encode(f.read()).decode()
response = ollama.chat(
model="llama3.2-vision:11b",
messages=[{
"role": "user",
"content": "What error is shown in this screenshot?",
"images": [image_data]
}]
)
print(response["message"]["content"])Drop a screenshot of an error message and ask: "What's causing this error and how do I fix it?"
Upload a PDF scan: "Extract all dates and amounts from this invoice."
Upload a business chart: "Describe the trend shown in this revenue chart."
Upload a website screenshot: "What UX improvements would you suggest for this form?"
With proper HIPAA setup: "Describe what you observe in this X-ray" — for research assistance, not diagnosis.
For audio understanding, combine Whisper (transcription) with any LLM:
# Install Whisper
pip install openai-whisper
# Transcribe audio
whisper meeting_recording.mp3 --model medium --output_format txt
# Then pipe transcript to Ollama
cat meeting_recording.txt | ollama run llama3.1 "Summarize the key decisions from this meeting transcript:"num_gpu is set to maximum in Ollama for vision modelsCan local vision models read text in images (OCR)? Yes — Qwen 2.5-VL 7B is particularly strong at OCR. It can read text from menus, receipts, whiteboards, and documents accurately.
How does local vision quality compare to GPT-4V? Gemma 3 27B and Qwen 2.5-VL 72B are competitive with GPT-4V for most tasks. For simple image description, even the 7–12B models are often sufficient.
Can I run video analysis locally? Not efficiently yet. Most tools process video as individual frames, which is slow. Frame-by-frame analysis at 1 frame/second is possible but impractical for long videos.
Do vision models work with PDF files? Indirectly — you need to convert PDF pages to images first, then send them to the vision model. Tools like pdf2image automate this.