Your Docker container can't see the GPU

Name: LLM Configurator — GPU VRAM Checker
Author: LLM Configurator

Linux

Written by Jakub Rusinowski · Last updated September 1, 2026

Founder, LLM Configurator — AI educator & workshop leader on local LLM deployment

The error

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

docker: Error response from daemon: unknown or invalid runtime name: nvidia.

bash: nvidia-smi: command not found   (inside the container)

Which one is it?

If you see	The cause is	Go to
"could not select device driver ... capabilities: [[gpu]]"	The NVIDIA Container Toolkit is not installed or not registered with Docker	Fix 1: install the toolkit and configure the runtime
"unknown or invalid runtime name: nvidia"	The toolkit is installed but Docker's daemon config does not know about it	Fix 2: nvidia-ctk runtime configure
`nvidia-smi: command not found` inside the container	The GPU was passed but the image contains no CUDA userspace	Fix 3: test with a CUDA base image
It works with sudo docker and not as your user	Rootless Docker needs its own toolkit configuration	Fix 4: configure the rootless runtime
AMD: /dev/kfd exists on the host but not inside the container	Device groups were not passed, or SELinux is blocking access	Fix 5: --group-add and setsebool

When you see it

The host is healthy. nvidia-smi prints a normal table, your GPU shows up, models run natively. Put the same workload in a container with --gpus all and Docker refuses to start it, or starts it and the container behaves as though there is no GPU in the machine.

You are moving a working native setup into Docker Compose and every GPU service fails to start.
You are running the Ollama or Open WebUI container image for the first time.
You reinstalled Docker, or moved from docker.io packages to Docker CE, and GPU access stopped.
You are on rootless Docker or Podman and following instructions written for rootful Docker.
You have a Radeon and are trying to pass /dev/kfd into a container.

What's actually going on

Docker containers get devices only when the runtime hands them over, and the default runtime knows nothing about GPUs. --gpus all is a request that must be serviced by the NVIDIA Container Toolkit: a set of hooks that inject the host driver's libraries and device nodes into the container at start. Without it, Docker parses the flag, finds no driver capable of providing [[gpu]], and refuses — which is exactly what the error says once you know to read it that way. The host driver is necessary and not sufficient. There is a second, quieter failure underneath: even with the toolkit working, an image with no CUDA userspace has no nvidia-smi to run, so a container can have the GPU and still look like it does not.

How to fix it

1. Install the NVIDIA Container Toolkit

Add NVIDIA's repository with a signed keyring and install the package. Check the current repository URL and keyring path against NVIDIA's installation guide before you run this — that path has changed more than once, and a stale copied-and-pasted line is a common reason the install appears to succeed and installs nothing. Verified against the NVIDIA Container Toolkit installation guide, 1 September 2026.

bash

# bash — Ubuntu / Debian
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

Did it work? The helper binary exists and reports a version. If nvidia-ctk is not found, the package did not install and the repository line is the thing to re-check.

nvidia-ctk --version
dpkg -l | grep nvidia-container-toolkit

2. Register the runtime with Docker and restart the daemon

Installing the toolkit does not tell Docker to use it — that is the second command, and skipping it produces the unknown or invalid runtime name: nvidia error. nvidia-ctk runtime configure edits /etc/docker/daemon.json for you; the daemon then has to be restarted before it re-reads that file.

bash

# bash
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# The canonical smoke test
docker run --rm --gpus all ubuntu nvidia-smi

Did it work? The container prints the same GPU table you get on the host. That single command passing means the whole chain — driver, toolkit, runtime, daemon — is working.

docker run --rm --gpus all ubuntu nvidia-smi
docker info | grep -i runtime

3. Test with an image that actually contains CUDA userspace

If --gpus all is accepted but the workload still cannot find a GPU, the image may simply have nothing to talk to it with. A plain ubuntu image works for nvidia-smi because the toolkit injects the host binary, but a framework needs CUDA libraries in the image. Test with an official CUDA base image to separate "the GPU is not being passed" from "my image is missing CUDA". For Ollama the published image already contains everything it needs.

bash

# bash — is the GPU visible to a real CUDA image?
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

# The Ollama container form
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 \
  --name ollama ollama/ollama

Did it work? The Ollama container reports GPU placement rather than CPU when a model is loaded.

docker exec ollama ollama run gemma3:1b "hi"
docker exec ollama ollama ps

4. Configure rootless Docker separately

Rootless Docker keeps its own daemon configuration under your home directory, so a runtime configure aimed at the system daemon does nothing for it. nvidia-ctk runtime configure --runtime=docker --config=$HOME/.config/docker/daemon.json targets the right file, and rootless mode also needs no-cgroups enabled in the container-toolkit config. This is the usual explanation for GPU access working under sudo docker and failing as your own user.

bash

# bash — rootless Docker
nvidia-ctk runtime configure --runtime=docker \
  --config=$HOME/.config/docker/daemon.json
systemctl --user restart docker

# Rootless also needs no-cgroups in the toolkit config:
sudo nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place

Did it work? The smoke test passes as your own user, with no sudo.

docker run --rm --gpus all ubuntu nvidia-smi

5. AMD containers: pass the device groups, and check SELinux

AMD does not use the NVIDIA toolkit. Containers need the device nodes passed explicitly along with the numeric group IDs that own them — numeric, because the group names inside the container will not match the host's. Get them from ls -lnd. On SELinux systems there is a second gate: Ollama's documentation notes SELinux can block container access to AMD GPU devices, and the fix is a single boolean.

⚠️ setsebool -P container_use_devices=1 permanently allows every container on this host to access device nodes, not just the one you are debugging. It is the documented fix, and it is a host-wide policy change — make it deliberately.

bash

# bash — find the numeric group IDs for the AMD device nodes
ls -lnd /dev/kfd /dev/dri /dev/dri/*

docker run --rm --device /dev/kfd --device /dev/dri \
  --group-add 44 --group-add 993 \
  rocm/dev-ubuntu-22.04 rocminfo

# SELinux hosts (Fedora, RHEL) only:
sudo setsebool -P container_use_devices=1

Did it work? rocminfo inside the container lists your GPU as an agent. If it lists only the CPU agent, the devices are still not reaching it.

docker run --rm --device /dev/kfd --device /dev/dri rocm/dev-ubuntu-22.04 rocminfo | grep -i "Marketing Name"

6. Podman equivalents

Podman uses CDI rather than a custom runtime. Generate a CDI specification for the GPUs once with nvidia-ctk cdi generate, then reference the device by name on each run. Regenerate the specification after a driver upgrade — a stale CDI file points at library paths that no longer exist, which produces a container that starts and then fails on the first CUDA call.

bash

# bash — Podman with NVIDIA
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
nvidia-ctk cdi list
podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi

# Podman with AMD
podman run --rm --device /dev/kfd --device /dev/dri \
  rocm/dev-ubuntu-22.04 rocminfo

Did it work? The CDI device appears in nvidia-ctk cdi list and the container prints the GPU table.

nvidia-ctk cdi list

7. Size the model for the GPU you are passing in Most common fix

Once the GPU reaches the container, the usual arithmetic applies and containers make it easier to get wrong: it is tempting to run several GPU services at once, and each loaded model holds its own VRAM. A container that had the whole card in testing may be sharing it with two others in Compose. Check what actually fits before you scale the stack up.

bash

# bash — what is holding VRAM across every container right now?
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

Check what fits your hardware — check what fits before you run several GPU containers at once
Open the VRAM checker →

If none of this worked

If the container sees the GPU but cannot reach Ollama on the host, that is a networking problem rather than a device problem and has its own page. On Windows, Docker Desktop with the WSL2 backend rides on the WSL driver plumbing, so fixing WSL fixes Docker there.

Include this when you report it

The exact docker run command and the full error
Output of nvidia-ctk --version and docker info | grep -i runtime
Whether Docker is rootful, rootless, or Podman
ls -lnd /dev/kfd /dev/dri /dev/dri/* for AMD problems

A model that fits most setups:

View model & requirements →

Frequently asked questions

Why is the host driver not enough for a container?

Because the container needs the driver's libraries and device nodes injected into its own filesystem and namespace, which is what the NVIDIA Container Toolkit does at start. Without it Docker has no driver able to satisfy the [[gpu]] capability and refuses to start the container.

What is the difference between the two error messages?

"could not select device driver ... [[gpu]]" means the toolkit is missing or not registered. "unknown or invalid runtime name: nvidia" means it is installed but Docker's daemon.json has not been configured — you skipped nvidia-ctk runtime configure or did not restart the daemon.

Why does it work with sudo but not as my user?

You are almost certainly on rootless Docker, which reads its own daemon config from your home directory. Run nvidia-ctk runtime configure against $HOME/.config/docker/daemon.json and set no-cgroups in the container-toolkit config.

How do I pass an AMD GPU into a container?

Pass /dev/kfd and /dev/dri as devices, and add the numeric group IDs owning them with --group-add — names differ between host and image, so use the numbers from ls -lnd. On SELinux hosts also set the container_use_devices boolean.

Do I need to redo any of this after a driver upgrade?

For Docker, usually not. For Podman, yes — regenerate the CDI specification, because it records library paths that a driver upgrade changes. A stale CDI file gives you a container that starts and then fails on its first CUDA call.