The Complete Ollama Bible: Install, Run & Master Local LLMs (2026)

Ollama is the single fastest way to get a real large language model running on your own computer. One install, one command — ollama run qwen3 — and you are chatting with a private AI that never sends a single token to the cloud. No API keys, no subscription, no data leaving your machine.

This is the complete guide. By the end you will understand not just the commands, but how Ollama actually works underneath — the model store, the quantization tags, the API server, the memory math — so you can pick the right model for your hardware, tune it for speed, plug it into other apps, and fix it when it breaks. It is written to be read start-to-finish by a beginner, or dipped into like a reference by someone who already has it installed.

Table of contents

• What Ollama is (and is not)
• Installing Ollama on Windows, macOS and Linux
• Your first model in 60 seconds
• How Ollama actually works under the hood
• Tokens, speed, and the numbers that matter
• Choosing a model that fits your hardware
• The model families you'll actually use
• Quantization: which tag should you pull?
• The complete command reference
• Modelfiles: build your own custom models
• The Ollama API and OpenAI compatibility
• Connecting a chat UI (Open WebUI and friends)
• Coding with Ollama: VS Code, Aider and agents
• Chatting with your documents (local RAG)
• Vision, images and multimodal models
• Automation recipes you can copy-paste
• Performance tuning for maximum speed
• Hardware-specific notes: Apple, AMD, Intel, multi-GPU
• Running Ollama as a home server
• Backups, disk management and moving your setup
• Troubleshooting the errors everyone hits
• Ollama vs LM Studio vs llama.cpp vs vLLM
• Frequently asked questions

What Ollama is (and is not) {#what-ollama-is}

Ollama is a model runner. Its job is to take an open-weight model — Meta's Llama, Alibaba's Qwen, DeepSeek, Google's Gemma, Microsoft's Phi — download it in a compressed, ready-to-run format, load it onto your GPU or CPU, and expose it to you as a chat prompt and to your programs as an API.

Think of it the way Docker thinks about containers. Docker gives you docker run nginx; Ollama gives you ollama run llama3.1. In both cases a big, fiddly thing (a server stack, a neural network) is packaged so that a single memorable command "just works." That analogy is not an accident — Ollama borrowed the mental model deliberately, right down to the Modelfile (its version of a Dockerfile).

Here is what Ollama gives you that raw model files do not:

• A curated library. The ollama.com/library catalog holds hundreds of models, each already converted to the GGUF format and quantized into several size options. You never hunt for the right file.
• Automatic hardware detection. Ollama figures out whether you have an NVIDIA GPU, an AMD GPU, an Apple Silicon chip, or only a CPU, and loads as many layers of the model onto the fastest device as will fit.
• A local server. Once installed, Ollama runs a background service with a REST API on http://localhost:11434. Every other tool in this guide — chat UIs, coding assistants, automation scripts — talks to that endpoint.
• Memory management. It loads models on demand, keeps them warm for a few minutes, and unloads them to free memory. You can run several models and it juggles them.

And here is what Ollama is not:

• It is not a model itself. Ollama is the engine; the intelligence comes from whichever model you pull. A weak model on Ollama is still weak.
• It is not the fastest possible server for heavy multi-user load. For a team hammering one endpoint with hundreds of simultaneous requests, vLLM is the production tool. Ollama is optimized for one person (or a small household) running models locally.
• It is not a cloud service. Nothing you type is transmitted anywhere. That is the entire point, and it is why businesses use it for privacy-safe document analysis.

If you want the graphical, click-to-download experience instead of a terminal, read the Complete LM Studio Bible — LM Studio is the GUI counterpart to Ollama. Many people run both. And for a side-by-side of the three big desktop runners, see Ollama vs LM Studio vs Jan.

Installing Ollama on Windows, macOS and Linux {#installing-ollama}

Ollama runs on all three major operating systems. Pick your platform.

Windows

1. Download the installer from ollama.com/download. 2. Run OllamaSetup.exe and click through. No admin rights or reboot required. 3. Ollama installs as a background service and adds a system-tray icon. Open a terminal (PowerShell or the new Windows Terminal) and you are ready.

ollama run qwen3

Windows requires Windows 10 22H2 or newer. If you have an NVIDIA or AMD GPU, install the latest vendor driver first — Ollama will detect and use it automatically.

macOS

1. Download the .dmg from ollama.com/download. 2. Drag Ollama into Applications and launch it once (this starts the background service). 3. Use the terminal exactly as on any other platform.

On Apple Silicon (M1 through M5), Ollama uses the Metal backend and the chip's unified memory — the RAM and VRAM are the same pool, which is why a MacBook with 32 GB or 64 GB of RAM can run models that would need an expensive multi-GPU rig on a PC. This is the single biggest reason Macs punch above their weight for local AI. See Best Mac for Local LLMs for buying advice.

Linux

The one-line installer handles everything, including NVIDIA/AMD detection:

curl -fsSL https://ollama.com/install.sh | sh

To run it as a persistent system service (recommended for a machine you leave on):

sudo systemctl enable ollama
sudo systemctl start ollama

Linux with a discrete NVIDIA GPU is the highest-performance option for local AI, and it is what most people rent when they rent a GPU by the hour. Make sure the proprietary NVIDIA driver and CUDA are installed; nvidia-smi should show your card before you start.

Docker

There is also an official image, useful for servers and reproducible setups:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Your first model in 60 seconds {#your-first-model}

The run command does everything at once: if the model is not already on disk, it downloads it, then loads it and drops you into an interactive chat.

ollama run qwen3

The first time, you will see a progress bar as the model downloads (a few gigabytes). After that it starts instantly. You get a >>> prompt — type a question, press Enter, and the model streams its answer.

A few things you can do inside the chat prompt:

• Type /bye to exit.
• Type /? to see slash commands.
• Type /set system You are a terse pirate. to change the model's persona on the fly.
• Paste multi-line text by wrapping it in triple quotes """.
• On multimodal models, drag an image path into the prompt to ask about a picture.

To just download a model without chatting (useful for pre-loading):

ollama pull llama3.1:8b

To see what you have downloaded:

ollama list

That is the whole loop: pull to get a model, run to talk to it, list to see them. Everything else in this guide is refinement.

How Ollama actually works under the hood {#how-ollama-works}

Understanding the plumbing turns Ollama from magic into a tool you can reason about. There are four moving parts.

1. The model store. When you pull a model, Ollama downloads GGUF files and stores them, by default, in ~/.ollama/models (on Windows, C:\Users\\.ollama\models). Models are content-addressed by hash, so two tags that share layers do not waste disk space. You can move this directory with the OLLAMA_MODELS environment variable — handy when your models outgrow your system drive. 2. The inference engine. Under the hood Ollama runs llama.cpp, the same battle-tested C++ engine that powers most of the local-LLM ecosystem. This is why Ollama supports GGUF, why quantization works the way it does, and why it runs on almost anything. Ollama is increasingly adding its own engine layer too, but llama.cpp remains the heart. 3. The model manager. This decides how many layers of the model fit in VRAM versus system RAM, loads them, and keeps the model "warm" in memory for 5 minutes after your last request (configurable) so the next question is instant. Run ollama ps to see what is loaded right now and whether it is on GPU or CPU. 4. The API server. A REST server listening on 11434. The terminal chat you used earlier is itself just a client of this API. So is every GUI. This separation — engine as a server, everything else as a client — is what makes Ollama so composable.

What a "model tag" really means

When you type ollama run llama3.1:8b, you are naming three things:

• llama3.1 — the model family.
• 8b — the tag, which usually encodes the parameter size and sometimes the quantization.
• An implied default quantization (Q4_K_M unless you specify otherwise), e.g. llama3.1:8b-instruct-q5_K_M.

Tags are how you pick exactly the trade-off you want between size, speed, and quality. But before you can choose well, it helps to understand what "speed" even means for an LLM.

Tokens, speed, and the numbers that matter {#tokens-and-speed}

Local-AI benchmarks throw around numbers like "92 t/s" and "180 ms TTFT." Here is what they mean and why they decide whether a model feels great or frustrating.

Tokens are the chunks a model reads and writes — roughly ¾ of a word each in English. "Hello, world!" is about four tokens. When you hear a model has a 128K context window, that is 128,000 tokens ≈ 96,000 words it can hold in mind at once. Your prompt, the conversation history, any documents you paste, and the model's reply all count against that budget.

There are two speeds that matter, and they measure different phases of a response:

Time to first token (TTFT) is the pause before the answer starts appearing. It is dominated by prefill — the model reading your prompt. Short prompt, short pause. Paste a 40-page PDF and the pause grows, because the model has to read all of it first. Decode speed (tokens/sec) is how fast the answer streams out once it starts. This is the number people quote. Crucially, decode speed is governed almost entirely by memory bandwidth — how fast your GPU can shuttle the model's weights in and out of memory — not by raw compute. That is why a RTX 5090 with 1,792 GB/s of bandwidth decodes far faster than a 4090 at 1,008 GB/s, and why the benchmarks page predicts speed from bandwidth. It is also why an Apple M-series chip, despite modest compute, is respectable: its unified memory has high bandwidth.

The practical upshot: 7–10 t/s is comfortably readable, 15–30 t/s feels snappy, and above 30 t/s the text arrives faster than you can read it. Below ~5 t/s it feels like waiting. If you drop below readable, your model has almost certainly spilled out of VRAM — the single most important thing to avoid, covered in the tuning section.

Choosing a model that fits your hardware {#choosing-a-model}

The number-one beginner mistake is pulling a model that is too big, watching it crawl at two tokens per second as it spills into system RAM, and concluding "local AI is slow." Local AI is fast — if the model fits in your VRAM.

The rule of thumb: a 4-bit quantized model needs roughly its parameter count in gigabytes of memory, plus overhead for context. An 8B model at Q4 wants about 6 GB; a 32B model wants about 20 GB; a 70B model wants about 42 GB. Add 1–3 GB for the context window.

You do not have to memorize any of this. The Hardware Analyzer reads your GPU and RAM and tells you exactly which models will run well, at what speed. If you know your model and just want the memory number, the VRAM Calculator does the arithmetic. And Can I Run It? answers the yes/no question per model. For the underlying theory, read VRAM Requirements Explained.

Good default models by hardware tier, mid-2026

New models land constantly. Browse the full, always-current Model Library for the latest, and read Choosing the Right LLM if you are unsure how to weigh reasoning vs. speed vs. creativity.

The model families you'll actually use {#model-families}

The Ollama library has hundreds of models, but a handful of families do 95% of the work in 2026. Here is the honest field guide — what each is for, and the exact tag to pull.

Qwen (Alibaba). The best all-round open family for local use. Qwen 3 introduced a hybrid thinking mode — type /think for deep chain-of-thought on hard problems, /no_think for fast answers. It comes in dense sizes (8B, 14B, 32B) and efficient MoE variants (30B-A3B). Its successor, Qwen 3.5, adds native vision and huge context. If you pull one model, pull qwen3:8b (small rigs) or qwen3:32b (24 GB+). Full details in the Complete Qwen Guide.

ollama run qwen3:8b        # best small all-rounder
ollama run qwen3:14b       # frontier reasoning on 12 GB
ollama run qwen3:32b       # flagship dense, needs 24 GB
ollama run qwen3:30b-a3b   # MoE: 14B quality at 7B speed

DeepSeek. The reasoning specialist. DeepSeek R1 and its distills think step-by-step before answering, which makes them exceptional at math, logic, and code — and slower, because that thinking costs tokens. The 14B distill fits a 12 GB card; the 32B distill is a 24 GB powerhouse. See the Complete DeepSeek Guide and our DeepSeek R1 vs GPT-4o comparison.

ollama run deepseek-r1:8b     # reasoning on a small card
ollama run deepseek-r1:14b    # the 12 GB sweet spot
ollama run deepseek-r1:32b    # 24 GB reasoning beast

Gemma (Google). Google's open family, strongest on multimodal (vision, and on small variants, audio) and famously good at small sizes. The Gemma 4 family runs from a phone-sized E2B up to a GPT-4-class 31B. Apache-style licensing makes it business-friendly. Read the Gemma 4 review.

ollama run gemma3:4b     # tiny, multimodal, laptop-friendly
ollama run gemma3:27b    # near-frontier on 16–24 GB

Llama (Meta). The family that started the open-weight wave. Still excellent general-purpose models with the broadest tooling support. llama3.1:8b remains a rock-solid default and the reference point most benchmarks compare against. Phi (Microsoft). Small models that punch far above their weight, trained on high-quality synthetic data. phi-4-mini is a great choice for weak hardware or CPU-only setups, and for running on a phone or Raspberry Pi. Mistral / Mixtral. Efficient European models with strong coding and a permissive Apache 2.0 license. Mistral Small 4 unified their reasoning, vision, and coding lines into one MoE that fits a 24 GB card. Specialists. For code, pull qwen3-coder (see Local Coding Assistant Setup). For embeddings (needed for RAG), pull nomic-embed-text or mxbai-embed-large. For long-document work at low VRAM, IBM's granite4.1 hybrid models are efficient and enterprise-certified.

The meta-point: families are not interchangeable. Reasoning models (DeepSeek R1, Qwen thinking mode) trade speed for correctness on hard problems; general chat models (Llama, base Qwen) are faster and friendlier for everyday tasks; multimodal models (Gemma, Qwen VL) see images. Match the family to the job. The Model Library tags every model by strength, and the Compare tool lets you put two side by side.

Quantization: which tag should you pull? {#quantization}

Quantization is the compression that makes local AI possible. A raw model stores each of its billions of weights as a 16-bit number. Quantization squeezes those down to 8, 5, 4, or even 2 bits each. The model gets dramatically smaller and faster, and — done well — barely less accurate.

You will see tags like these on ollama.com. Here is what they mean and when to use them:

The takeaway most people need: use Q4_K_M. When you run ollama run qwen3:8b, that is already what you get. Only reach for a higher quant if the model comfortably fits with room left, or a lower quant if you are trying to squeeze a larger, smarter model onto a small card — a bigger model at Q3 usually beats a smaller model at Q8. For the full theory, see Quantization Explained and our deep dive LLM Quantization Explained.

To pull a specific quant, append it to the tag:

ollama pull qwen3:8b-instruct-q5_K_M

The complete command reference {#command-reference}

Everything Ollama does from the terminal, in one place.

# --- Getting and running models ---
ollama run <model>            # download if needed, then chat
ollama run <model> "prompt"   # one-shot: answer and exit (great for scripts)
ollama pull <model>           # download only, no chat
ollama list                   # list installed models and their sizes
ollama ps                     # show running models + whether on GPU or CPU
ollama stop <model>           # unload a model from memory now
ollama rm <model>             # delete a model from disk
ollama cp <src> <dest>        # copy/rename a model

# --- Inspecting ---
ollama show <model>           # license, parameters, template, context length
ollama show <model> --modelfile   # dump the Modelfile

# --- Custom models ---
ollama create <name> -f Modelfile   # build a model from a Modelfile
ollama push <name>            # push to your ollama.com account

# --- The server ---
ollama serve                  # run the API server in the foreground
ollama --version

A pattern worth knowing: ollama run with a quoted prompt is a one-shot call that prints the answer and exits. That makes Ollama usable straight from shell scripts and pipes:

# summarize a file from the command line
cat notes.txt | ollama run llama3.1:8b "Summarize the following in 3 bullets:"

# generate a commit message from a diff
git diff --staged | ollama run qwen3:8b "Write a concise git commit message for this diff:"

Modelfiles: build your own custom models {#modelfiles}

A Modelfile is a small recipe that bakes a base model plus your settings into a new, reusable model. It is how you create a permanently-configured assistant — a coding helper, a JSON-only extractor, a specific persona — without repeating yourself.

Create a file called Modelfile:

FROM qwen3:8b

# Creativity: 0 = deterministic, 1+ = wild
PARAMETER temperature 0.4
# Context window in tokens
PARAMETER num_ctx 8192
# Stop generating at these strings
PARAMETER stop "<|im_end|>"

# The system prompt — the model's standing instructions
SYSTEM """
You are a senior Python engineer. You answer with working code first,
then a one-paragraph explanation. You never apologize. If a request is
ambiguous you state your assumption and proceed.
"""

Then build and run it:

ollama create py-helper -f Modelfile
ollama run py-helper

The key Modelfile instructions:

• FROM — the base model (a library tag, or a path to a local GGUF file you downloaded yourself).
• PARAMETER — inference settings: temperature, top_p, top_k, num_ctx (context length), repeat_penalty, stop, seed, and more.
• SYSTEM — the system prompt. This is the highest-leverage line; see System Prompts & Personas 101 for how to write good ones.
• TEMPLATE — the exact chat format the model expects (usually inherited from the base; only touch it for custom base models).
• ADAPTER — apply a LoRA fine-tune on top of the base. This is how you deploy a model you trained yourself — see Fine-Tune Your First LLM in 1 Hour and Convert a Fine-Tune to GGUF.

Modelfiles are the bridge between "I use models other people made" and "I ship my own." If you have a GGUF from Hugging Face that is not in the Ollama library, a two-line Modelfile (FROM ./model.gguf) imports it.

Sampling parameters explained

The PARAMETER lines control how the model chooses its next token. You do not need to touch most of them, but understanding the big three saves a lot of "why is the output weird?" frustration. You can set them permanently in a Modelfile, or per-session with /set parameter .

The one to reach for is temperature. If a model is rambling or making things up, lower it. If it is boring and repetitive, raise it. For anything where correctness matters — extracting data, writing code, answering factual questions — run cold (0.1–0.3). For brainstorming and fiction, run warm (0.9–1.2). Setting seed plus a fixed temperature makes outputs reproducible, which is essential when you are testing prompts.

Modelfiles are also composable: you can ollama create several personas from the same base model — a terse coder, a friendly tutor, a JSON extractor — each with its own system prompt and temperature, and switch between them by name.

The Ollama API and OpenAI compatibility {#the-api}

This is where Ollama stops being a chat toy and becomes infrastructure. The server on 11434 exposes two API styles. Since version 0.35 it also serves a third, specialised endpoint, /v1/systemone, for decision models that return a probability per option rather than text; the setup guide walks through it.

The native API

# Simple generation
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

# Chat with message history
curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

The OpenAI-compatible API

This is the important one. Ollama speaks the exact same protocol as the OpenAI API at /v1. That means any tool, library, or app built for OpenAI works with your local model by changing one line — the base URL — and using any dummy API key.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",          # required by the library, ignored by Ollama
)

resp = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Write a haiku about local AI."}],
)
print(resp.choices[0].message.content)

Because of this compatibility, you can point LangChain, LlamaIndex, continue.dev, Aider, countless VS Code extensions, and thousands of other OpenAI-based projects at your local Ollama with a one-line change. Your cloud bill goes to zero and your data stays home. This is also how you build a drop-in local API server for a whole team.

Ollama also serves an embeddings endpoint (/api/embeddings), which is what powers local Retrieval-Augmented Generation — chatting with your own documents. Combine an embedding model like nomic-embed-text with a chat model and you have a fully local RAG pipeline.

Message roles and the chat template

When you use /api/chat, you send a list of messages, each with a role. Getting these right is the difference between a well-behaved assistant and a confused one:

• system — standing instructions that shape the model's whole behavior: its persona, rules, and output format. Sent once, at the start. This is the highest-leverage message you can write.
• user — what the human says.
• assistant — what the model previously replied. You resend the growing history each turn so the model "remembers" the conversation; Ollama is stateless per request, so the context is the messages you send.
• tool — results handed back from a function the model asked to call, for agentic/tool-use workflows.

Under the hood, Ollama wraps these messages in the model's exact chat template — the special tokens (like <|im_start|>) each model family was trained to expect. You almost never touch the template directly; Ollama applies the correct one automatically. It matters only when you import a raw GGUF whose template Ollama cannot infer, in which case you specify TEMPLATE in your Modelfile. If a home-grown model produces garbled or never-ending output, a wrong template is the usual culprit.

Connecting a chat UI (Open WebUI and friends) {#chat-ui}

The terminal is fine, but most people want a polished chat window with history, folders, file uploads, and multiple models. Ollama pairs with every major local UI.

The most popular partner is Open WebUI — a self-hosted, ChatGPT-style interface that auto-detects your Ollama models. It runs in Docker and gives you multi-user accounts, document chat (RAG), web search, and image generation. It is so central to the local-AI stack that we wrote a whole Complete Open WebUI Guide for it.

The fastest way to try it:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000, and every model you have pulled in Ollama appears in the dropdown. Other good options include Jan, LM Studio (which can also act as a client), Msty, and AnythingLLM. Our roundup Open WebUI vs LM Studio vs Jan compares them.

Coding with Ollama: VS Code, Aider and agents {#coding}

One of the highest-value uses of a local model is as a coding assistant — autocomplete, "explain this function," "write a test," "fix this bug" — running entirely on your machine, with your proprietary code never leaving it. Because Ollama speaks the OpenAI protocol, the popular coding tools all connect in minutes.

Continue (VS Code / JetBrains). The most popular open-source Copilot alternative. Install the Continue extension, open its config, and point it at Ollama:

{
  "models": [
    { "title": "Qwen Coder", "provider": "ollama", "model": "qwen3-coder:latest" }
  ],
  "tabAutocompleteModel": { "provider": "ollama", "model": "qwen3-coder:latest" }
}

Aider (terminal pair-programmer). Aider edits your git repo through conversation and commits as it goes:

export OLLAMA_API_BASE=http://127.0.0.1:11434
aider --model ollama_chat/qwen3-coder

For coding, three things matter: pull a code-tuned model (qwen3-coder is the strongest local option in 2026), give it a big context (num_ctx 16384 or more so it can see whole files), and prefer a model that fits your VRAM at that context — spilling kills the fast autocomplete loop. The dedicated Local Coding Assistant Setup and Coding & Debugging Workflow guides go deep, and the Coding hub covers agentic setups.

Chatting with your documents (local RAG) {#rag}

Retrieval-Augmented Generation (RAG) is how you get a model to answer questions about your files — a folder of PDFs, a wiki, a codebase — without fine-tuning. The mechanism: split your documents into chunks, convert each chunk to a numeric vector with an embedding model, store the vectors, and at question time retrieve the most relevant chunks and hand them to the chat model as context.

The simplest way to get RAG is to use Open WebUI, which has document upload and a knowledge base built in — no code. For a DIY pipeline, pull an embedding model and wire it up:

ollama pull nomic-embed-text

import ollama
# turn a chunk of text into a vector
v = ollama.embeddings(model="nomic-embed-text", prompt="Ollama runs models locally.")
print(len(v["embedding"]))   # e.g. 768 numbers

From there, a vector store (Chroma, FAISS, or Postgres + pgvector) holds the vectors and returns the closest matches. The full build — chunking strategy, retrieval, and prompt assembly — is in Intro to RAG: Chat with Docs and the advanced Complete Local RAG Pipeline. RAG is the single best way to make a local model useful for document analysis.

Vision, images and multimodal models {#vision}

Several models in the Ollama library see. Multimodal models accept an image alongside your text and can describe it, read text out of it (OCR), interpret charts, or reason about a screenshot. Good choices: gemma3 (all sizes), qwen2.5vl, and Llama vision variants.

From the terminal, reference an image path in your prompt:

ollama run gemma3:4b
>>> What's in this image? ./invoice.png

From the API, pass base64 image data in the images array:

import ollama
res = ollama.chat(model="gemma3:4b", messages=[{
    "role": "user",
    "content": "Extract the total and due date from this invoice.",
    "images": ["./invoice.png"],
}])
print(res["message"]["content"])

This unlocks receipt/invoice parsing, screenshot Q&A, handwriting recognition, and "describe this photo" — all offline. See Multimodal LLMs (Vision + Text) for the full rundown and model comparison.

Automation recipes you can copy-paste {#automation}

Because ollama run "" is a one-shot command and the API is trivial to call, Ollama slots into scripts, cron jobs, and pipelines. A few practical recipes:

# 1. Batch-summarize every markdown file in a folder
for f in ./notes/*.md; do
  echo "## Summary of $f" >> digest.md
  cat "$f" | ollama run qwen3:8b "Summarize in 3 bullets:" >> digest.md
done

# 2. Classify incoming text into a fixed set of labels
echo "My package never arrived and support won't reply." \
  | ollama run llama3.1:8b "Classify as one of: billing, shipping, technical, other. Reply with only the label."

# 3. Extract structured JSON from messy text
cat email.txt | ollama run qwen3:8b 'Extract sender, date, and action items as JSON. Output only JSON.'

For reliable machine-readable output, use the API's structured output / format option, which constrains the model to valid JSON matching a schema — far more dependable than asking nicely. The Reliable JSON & Structured Output guide shows the exact request format. String several of these calls together and you have a fully local automation that never risks leaking data to a third party — the core appeal explored in local LLM use cases for 2026.

Performance tuning for maximum speed {#performance-tuning}

If your model fits in VRAM and you still want more speed, or it almost fits and you want to make it fit, these are the knobs.

Keep the whole model on the GPU

The single biggest factor is whether every layer is on the GPU. Run ollama ps — if it says 100% GPU, you are golden. If it shows a CPU percentage, part of the model spilled to system RAM and that portion runs an order of magnitude slower. Fixes: pull a smaller quant, reduce the context window, or close other VRAM-hungry apps.

Useful environment variables

Set these before ollama serve (or in the service config):

# Store models on a bigger drive
export OLLAMA_MODELS=/mnt/big-disk/ollama-models

# Keep models loaded longer (default 5m). -1 = forever, 0 = unload immediately
export OLLAMA_KEEP_ALIVE=30m

# Allow more than one model / more parallel requests
export OLLAMA_NUM_PARALLEL=4
export OLLAMA_MAX_LOADED_MODELS=2

# Speed + memory wins on most modern GPUs
export OLLAMA_FLASH_ATTENTION=1
# Quantize the KV cache to halve context memory (q8_0 is near-lossless)
export OLLAMA_KV_CACHE_TYPE=q8_0

# Expose the server to your LAN (see the home-server section)
export OLLAMA_HOST=0.0.0.0:11434

Flash attention and KV-cache quantization together can roughly halve the memory your context window consumes, letting you run longer contexts or bigger models. On a card like the RTX 5090 these are close to free wins.

The context window trade-off

num_ctx sets how many tokens the model can "see." Bigger context = more memory and slightly slower. Ollama historically defaulted to a small context (2048) for compatibility; if you are doing document analysis, bump it:

ollama run qwen3:8b
>>> /set parameter num_ctx 16384

Or bake it into a Modelfile. Just remember the context lives in VRAM — a huge context on a full GPU will force a spill. Read Context Windows Explained for the full picture of how context length interacts with memory and speed.

Hardware-specific notes: Apple, AMD, Intel, multi-GPU {#hardware-notes}

Ollama runs on far more than NVIDIA cards, but each platform has a couple of things worth knowing.

NVIDIA (the default). The smoothest experience. Install the current driver; Ollama bundles the CUDA it needs. Everything in this guide assumes NVIDIA unless noted. The RTX 5090, RTX 4090, and RTX 5080 are the enthusiast picks; the best GPU under $500 covers budget options. Compare any two cards on the GPU benchmarks page. Apple Silicon. M1–M5 chips use the Metal backend automatically. The magic is unified memory: system RAM doubles as VRAM, so a 64 GB MacBook can load models that would need two 24 GB PC GPUs. The catch is memory bandwidth scales with the chip tier — a Max or Ultra chip decodes much faster than a base M-chip. For buying advice see Best Mac for Local LLMs. AMD. Supported via ROCm on Linux and Windows for recent Radeon cards (RX 7000/9000 series and several others). The RX 7900 XTX with 24 GB is the standout value pick. Make sure your card is on AMD's ROCm support list; older cards may fall back to CPU. Intel. Arc GPUs and recent integrated graphics have growing support; check the current Ollama release notes, as this is the fastest-moving backend. Multiple GPUs. Ollama automatically splits a model across all visible GPUs, letting you run, say, a 70B model on two 24 GB cards. It is model parallelism, not a speed multiplier — two cards let you run bigger models, not necessarily faster ones. Control which GPUs are used with CUDA_VISIBLE_DEVICES. The trade-offs (and when a second card is worth it) are in Multi-GPU Inference: Worth It?. No GPU at all. Ollama runs on CPU using system RAM. Stick to 1B–4B models and enable as many threads as you have cores. It is genuinely usable for small models — see Do you need an expensive PC for local AI?.

Running Ollama as a home server {#home-server}

One of the best local-AI setups is a single always-on machine (a spare PC, a mini PC, a Mac) running Ollama, which every device in your house then talks to — laptop, phone, tablet.

Set OLLAMA_HOST=0.0.0.0:11434 so the server accepts connections from your network, restart Ollama, and point clients at http://:11434. For access from outside your home safely, do not expose 11434 to the internet directly — put it behind Tailscale (a zero-config private network) or a reverse proxy with authentication. Our Personal AI Home Server guide walks through the Tailscale setup step by step, and you can even reach it from your phone.

Backups, disk management and moving your setup {#backups}

Models are large, and they add up fast — a serious library can eat 100+ GB. A few housekeeping habits keep things sane.

See what you have and reclaim space. ollama list shows every model and its size; ollama rm deletes ones you no longer use. Because layers are content-addressed, deleting one tag will not break another that shares layers. Move the model store to a bigger drive. Set OLLAMA_MODELS to a path on your large disk and restart Ollama. On Windows this is a system environment variable; on Linux, add it to the systemd service or your shell profile. This is the standard fix for "my C: drive is full." Back up or migrate. The entire store lives under ~/.ollama (models plus your custom Modelfiles). Copy that directory to move a whole setup to a new machine — no re-downloading. The Move Your Setup to a New Computer guide covers the edge cases. Where the logs are. When something misbehaves, the server log is your friend. On macOS: ~/.ollama/logs/server.log. On Linux (systemd): journalctl -u ollama. On Windows: %LOCALAPPDATA%\Ollama\server.log. Set OLLAMA_DEBUG=1 for verbose output that shows exactly how layers were split across GPU and CPU — invaluable for diagnosing a spill.

Troubleshooting the errors everyone hits {#troubleshooting}

For model-specific quirks and deeper diagnostics, the Troubleshooting hub has dedicated pages.

Ollama vs LM Studio vs llama.cpp vs vLLM {#comparison}

Ollama is not the only runner. Here is where it sits.

Rule of thumb: start with Ollama or LM Studio. Use LM Studio if you want to click; use Ollama if you want to script or serve. Graduate to vLLM only when you are serving a real crowd. See our full comparison of when to use vLLM vs Ollama.

Frequently asked questions {#faq}

Is Ollama free?

Yes, completely. Ollama is MIT-licensed open source, and the open-weight models it runs are free to download. Your only cost is the electricity to run your own hardware — a fraction of a cloud API bill. See why local LLMs are cheaper and more secure and the Cost Calculator to compare against cloud APIs.

Does Ollama send my data anywhere?

No. Inference happens entirely on your machine. The only network traffic is downloading models from the library (which you can mirror for fully offline use). This is why it is trusted for private document analysis and regulated business use.

Can I run Ollama without a GPU?

Yes. Ollama runs on CPU alone — just stick to small models (1B–4B) and expect slower speeds. A modern laptop CPU handles a 3B model at a readable pace. See Running on Laptops & Raspberry Pi and Do you need an expensive PC for local AI?.

What is the difference between ollama pull and ollama run?

pull only downloads. run downloads if needed, then starts a chat. run with a quoted string does a one-shot answer and exits.

How do I update a model or Ollama itself?

Re-pull the model tag (ollama pull qwen3) to get the latest build. Update Ollama by re-running the installer or curl ... | sh on Linux.

Which model is best?

There is no single best — it depends on your hardware and task. For most people in 2026, Qwen 3 is the best all-rounder that fits consumer GPUs, DeepSeek R1 for hard reasoning, and Gemma for multimodal. Use the Model Library and Hardware Analyzer to decide, or read Who is winning the local LLM race in 2026.

Can Ollama do images, vision, or voice?

Yes for vision — multimodal models (Gemma, Qwen VL, Llama vision) accept image input; drag an image into the chat or send it via the API. For voice, pair Ollama with Whisper (speech-to-text) and a TTS engine as in the Offline Voice Assistant guide.

Can I run more than one model at once?

Yes. Set OLLAMA_MAX_LOADED_MODELS higher than 1 and Ollama keeps several models resident, loading and unloading as memory allows. OLLAMA_NUM_PARALLEL controls how many simultaneous requests one model handles. This is how a home server serves a chat model and an embedding model at the same time.

Can I import a GGUF I downloaded from Hugging Face?

Yes. Write a one-line Modelfile — FROM ./the-model.gguf — and run ollama create myname -f Modelfile. Ollama also supports pulling many community GGUFs directly with ollama run hf.co//.

Does Ollama work fully offline?

Once a model is downloaded, yes — inference needs no network at all. Pre-pull your models while online and you can run on a plane, in an air-gapped lab, or anywhere with no connection. The air-gapped setup guide covers a truly disconnected deployment.

Can I use Ollama commercially?

Ollama itself is MIT-licensed, so yes. But each model has its own license — Qwen, Gemma, DeepSeek, Mistral, and Granite are permissively licensed (Apache 2.0 / MIT), while some Llama models carry community-license conditions. Always check ollama show for the license before shipping a product. Our Private AI for Business and open LLM licenses resources cover the details.

How do I uninstall Ollama?

On macOS, delete the app and ~/.ollama. On Windows, use Add/Remove Programs and delete %USERPROFILE%\.ollama. On Linux, stop the service, remove the binary, and delete ~/.ollama and /usr/share/ollama.

Ollama or LM Studio — which should a beginner start with?

If a terminal makes you nervous, start with LM Studio — it is point-and-click. If you are comfortable with a command line or plan to script and serve, start with Ollama. They read the same GGUF models, so you can switch or run both. The side-by-side comparison helps you choose.

Capstone: build a private research assistant in 15 minutes

Let us tie the whole guide together with something real — a local assistant that reads your PDFs and answers questions about them, entirely offline. This uses five things you now understand: pulling models, quantization, the API, a Modelfile, and RAG.

Step 1 — pull a chat model and an embedding model. On a 12–16 GB card, qwen3:14b is a strong reasoner; on 8 GB, use qwen3:8b. The embedding model is tiny.

ollama pull qwen3:14b
ollama pull nomic-embed-text

Step 2 — give it a personality with a Modelfile. We want careful, cited answers, not confident guesses.

FROM qwen3:14b
PARAMETER temperature 0.2
PARAMETER num_ctx 16384
SYSTEM """
You are a meticulous research assistant. Answer only from the provided
context. If the answer is not in the context, say "I don't see that in
the documents." Quote the relevant passage, then explain in plain English.
"""

ollama create researcher -f Modelfile

The low temperature keeps it factual, the large context lets it hold several document chunks at once, and the system prompt forbids it from making things up — the classic RAG guardrail.

Step 3 — the easy path. Point Open WebUI at your Ollama, upload your PDFs into a "knowledge" collection, select the researcher model, and ask away. Open WebUI handles the chunking, embedding, and retrieval for you. This is genuinely a 10-minute, zero-code setup. Step 4 — the code path. If you would rather script it, the loop is: embed each document chunk with nomic-embed-text, store the vectors, embed the question, retrieve the closest chunks, and send them plus the question to researcher via the OpenAI-compatible endpoint. The full working code is in the Complete Local RAG Pipeline.

That is a capable, private document assistant — the kind of thing companies pay per-seat cloud fees for — running on a single consumer GPU with nothing leaving your machine. Swap in DeepSeek R1 for heavier reasoning, or a vision model to also read scanned pages. From here you can grow it into an autonomous agent or a team-wide local API server.

A quick glossary

Every term in this guide, in one place:

• GGUF — the file format Ollama uses to store quantized models. One file holds the weights plus metadata.
• Quantization — compressing model weights to fewer bits (e.g. 4-bit Q4_K_M) to shrink size and boost speed.
• VRAM — the memory on your GPU. The model must fit here to run fast. On Apple Silicon, unified RAM plays this role.
• Token — a chunk of text (~¾ of a word) that models read and write. Speed is measured in tokens/sec.
• Context window — how many tokens the model can consider at once (its short-term memory), set by num_ctx.
• TTFT — time to first token; the pause before the answer starts, driven by prompt length.
• Modelfile — a recipe that bakes a base model plus settings and a system prompt into a reusable custom model.
• System prompt — the model's standing instructions; the biggest lever on its behavior.
• MoE (Mixture of Experts) — an architecture where only part of the model activates per token, giving big-model quality at small-model speed. See MoE Models Explained.
• RAG — Retrieval-Augmented Generation; feeding the model relevant chunks of your own documents at question time.
• Embedding — a numeric vector representing the meaning of text, used to find relevant chunks for RAG.

Where to go next

You now know how to install Ollama, pick a model that fits, choose a quant, script the API, build custom Modelfiles, tune for speed, and troubleshoot. The natural next steps:

• Give it a face: set up the Complete Open WebUI Guide for a ChatGPT-style interface.
• Prefer clicking? The Complete LM Studio Bible covers the GUI route.
• Pick your model: read the Complete Qwen Guide and Complete DeepSeek Guide.
• Pick your hardware: the Complete RTX 5090 Guide and Best GPU Buyer's Guide.
• Chat with your files: build a local RAG pipeline.

Ollama is the foundation. Everything else in local AI plugs into it.

---

Check which models your machine can run → Hardware Analyzer Browse every open model, always current → Model Library Compare local running costs to cloud APIs → Cost Calculator