Docker + Ollama: Production Deployment Guide (2026)

Written by Jakub Rusinowski · Last updated July 21, 2026

Running Ollama in Docker gives you reproducible, isolated, easy-to-update AI infrastructure. This guide covers everything from a single-server setup to a multi-GPU deployment with a load balancer.

In This Guide

• Why Docker for Ollama?
• Prerequisites
• Step 1: Install NVIDIA Container Toolkit (GPU passthrough)
• Step 2: Run Ollama with GPU
• Step 3: Pull Models Inside Docker
• Step 4: Production docker-compose.yml
• Environment Variables for Tuning
• Multi-GPU Setup

Running Ollama in Docker gives you reproducible, isolated, easy-to-update AI infrastructure. This guide covers everything from a single-server setup to a multi-GPU deployment with a load balancer.

Why Docker for Ollama?

• Reproducibility: Same environment on dev and prod
• Easy updates: docker pull updates Ollama without uninstalling
• Isolation: AI dependencies don't pollute your system
• Orchestration: Combine with docker-compose for full AI stack
• GPU passthrough: Full NVIDIA/AMD GPU support via container runtime

Prerequisites

• Docker Engine 24+ (or Docker Desktop)
• NVIDIA Container Toolkit (for GPU support)
• An NVIDIA GPU with 6+ GB VRAM (or AMD with ROCm support)

Step 1: Install NVIDIA Container Toolkit (GPU passthrough)

# Ubuntu/Debian
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Step 2: Run Ollama with GPU

docker run -d \
  --gpus all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  --restart unless-stopped \
  ollama/ollama

The -v ollama:/root/.ollama mounts a named volume so model files persist across container restarts.

Step 3: Pull Models Inside Docker

# Pull a model
docker exec -it ollama ollama pull llama3.1:8b

# List available models
docker exec -it ollama ollama list

# Run a test prompt
docker exec -it ollama ollama run llama3.1:8b "What is VRAM?"

Step 4: Production docker-compose.yml

Create a full AI stack with Ollama + Open WebUI:

version: "3.8"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "11434:11434"
    environment:
      - OLLAMA_NUM_PARALLEL=4       # Allow 4 concurrent requests
      - OLLAMA_MAX_LOADED_MODELS=2  # Keep 2 models in VRAM simultaneously
      - OLLAMA_KEEP_ALIVE=24h       # Keep models loaded for 24 hours

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    volumes:
      - webui_data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    depends_on:
      - ollama

volumes:
  ollama_data:
  webui_data:

Start everything:

docker-compose up -d

Environment Variables for Tuning

Variable	Default	Description
OLLAMA_NUM_PARALLEL	1	Concurrent inference requests
OLLAMA_MAX_LOADED_MODELS	1	Models kept in VRAM at once
OLLAMA_KEEP_ALIVE	5m	How long to keep a model in VRAM after use
OLLAMA_MAX_QUEUE	512	Max queued requests before rejecting
OLLAMA_NUM_GPU	all	Number of GPUs to use

Multi-GPU Setup

For multiple GPUs, Ollama automatically uses all available GPUs for a single model via tensor parallelism:

docker run -d --gpus all \
  -e OLLAMA_NUM_GPU=2 \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama

With dual RTX 4090s (48 GB combined VRAM), you can run Llama 3.3 70B Q4 at ~55 tokens/second.

Health Check & Monitoring

# Check if Ollama is healthy
curl http://localhost:11434/api/health

# Monitor GPU usage
watch -n1 nvidia-smi

# View container logs
docker logs -f ollama

# Check API response
curl http://localhost:11434/api/tags

Exposing Ollama Securely (Nginx + SSL)

nginx.conf for reverse proxy with basic auth:

server {
    listen 443 ssl;
    server_name your-ai.example.com;

    ssl_certificate /etc/letsencrypt/live/your-ai.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your-ai.example.com/privkey.pem;

    location / {
        auth_basic "Restricted";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://localhost:11434;
        proxy_set_header Host $host;
        proxy_read_timeout 600s;
    }
}

Updating Ollama

docker pull ollama/ollama:latest
docker-compose down
docker-compose up -d

Model files in the ollama_data volume are preserved — no need to re-pull models.

→ Back to all guides | → Check hardware compatibility

← All Guides | Check GPU Compatibility