Won't start or won't see your GPULinuxLast updated September 1, 2026

Your Docker container can't see the GPU

nvidia-smi works on the host, fails inside the container

Written by Jakub Rusinowski

The error

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

docker: Error response from daemon: unknown or invalid runtime name: nvidia.

bash: nvidia-smi: command not found   (inside the container)
Which one is it?
If you seeThe cause isGo to
"could not select device driver ... capabilities: [[gpu]]"The NVIDIA Container Toolkit is not installed or not registered with DockerFix 1: install the toolkit and configure the runtime
"unknown or invalid runtime name: nvidia"The toolkit is installed but Docker's daemon config does not know about itFix 2: nvidia-ctk runtime configure
nvidia-smi: command not found inside the containerThe GPU was passed but the image contains no CUDA userspaceFix 3: test with a CUDA base image
It works with sudo docker and not as your userRootless Docker needs its own toolkit configurationFix 4: configure the rootless runtime
AMD: /dev/kfd exists on the host but not inside the containerDevice groups were not passed, or SELinux is blocking accessFix 5: --group-add and setsebool

When you see it

The host is healthy. nvidia-smi prints a normal table, your GPU shows up, models run natively. Put the same workload in a container with --gpus all and Docker refuses to start it, or starts it and the container behaves as though there is no GPU in the machine.

  • You are moving a working native setup into Docker Compose and every GPU service fails to start.
  • You are running the Ollama or Open WebUI container image for the first time.
  • You reinstalled Docker, or moved from docker.io packages to Docker CE, and GPU access stopped.
  • You are on rootless Docker or Podman and following instructions written for rootful Docker.
  • You have a Radeon and are trying to pass /dev/kfd into a container.

What's actually going on

Docker containers get devices only when the runtime hands them over, and the default runtime knows nothing about GPUs. --gpus all is a request that must be serviced by the NVIDIA Container Toolkit: a set of hooks that inject the host driver's libraries and device nodes into the container at start. Without it, Docker parses the flag, finds no driver capable of providing [[gpu]], and refuses — which is exactly what the error says once you know to read it that way. The host driver is necessary and not sufficient. There is a second, quieter failure underneath: even with the toolkit working, an image with no CUDA userspace has no nvidia-smi to run, so a container can have the GPU and still look like it does not.

How to fix it

  1. 1

    Install the NVIDIA Container Toolkit

    Add NVIDIA's repository with a signed keyring and install the package. Check the current repository URL and keyring path against NVIDIA's installation guide before you run this — that path has changed more than once, and a stale copied-and-pasted line is a common reason the install appears to succeed and installs nothing. Verified against the NVIDIA Container Toolkit installation guide, 1 September 2026.

    bash
    # bash — Ubuntu / Debian
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
    Did it work?The helper binary exists and reports a version. If nvidia-ctk is not found, the package did not install and the repository line is the thing to re-check.
    bash
    nvidia-ctk --version
dpkg -l | grep nvidia-container-toolkit
  2. 2

    Register the runtime with Docker and restart the daemon

    Installing the toolkit does not tell Docker to use it — that is the second command, and skipping it produces the unknown or invalid runtime name: nvidia error. nvidia-ctk runtime configure edits /etc/docker/daemon.json for you; the daemon then has to be restarted before it re-reads that file.

    bash
    # bash
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# The canonical smoke test
docker run --rm --gpus all ubuntu nvidia-smi
    Did it work?The container prints the same GPU table you get on the host. That single command passing means the whole chain — driver, toolkit, runtime, daemon — is working.
    bash
    docker run --rm --gpus all ubuntu nvidia-smi
docker info | grep -i runtime
  3. 3

    Test with an image that actually contains CUDA userspace

    If --gpus all is accepted but the workload still cannot find a GPU, the image may simply have nothing to talk to it with. A plain ubuntu image works for nvidia-smi because the toolkit injects the host binary, but a framework needs CUDA libraries in the image. Test with an official CUDA base image to separate "the GPU is not being passed" from "my image is missing CUDA". For Ollama the published image already contains everything it needs.

    bash
    # bash — is the GPU visible to a real CUDA image?
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

# The Ollama container form
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 \
  --name ollama ollama/ollama
    Did it work?The Ollama container reports GPU placement rather than CPU when a model is loaded.
    bash
    docker exec ollama ollama run gemma3:1b "hi"
docker exec ollama ollama ps
  4. 4

    Configure rootless Docker separately

    Rootless Docker keeps its own daemon configuration under your home directory, so a runtime configure aimed at the system daemon does nothing for it. nvidia-ctk runtime configure --runtime=docker --config=$HOME/.config/docker/daemon.json targets the right file, and rootless mode also needs no-cgroups enabled in the container-toolkit config. This is the usual explanation for GPU access working under sudo docker and failing as your own user.

    bash
    # bash — rootless Docker
nvidia-ctk runtime configure --runtime=docker \
  --config=$HOME/.config/docker/daemon.json
systemctl --user restart docker

# Rootless also needs no-cgroups in the toolkit config:
sudo nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place
    Did it work?The smoke test passes as your own user, with no sudo.
    bash
    docker run --rm --gpus all ubuntu nvidia-smi
  5. 5

    AMD containers: pass the device groups, and check SELinux

    AMD does not use the NVIDIA toolkit. Containers need the device nodes passed explicitly along with the numeric group IDs that own them — numeric, because the group names inside the container will not match the host's. Get them from ls -lnd. On SELinux systems there is a second gate: Ollama's documentation notes SELinux can block container access to AMD GPU devices, and the fix is a single boolean.

    Careful: setsebool -P container_use_devices=1 permanently allows every container on this host to access device nodes, not just the one you are debugging. It is the documented fix, and it is a host-wide policy change — make it deliberately.
    bash
    # bash — find the numeric group IDs for the AMD device nodes
ls -lnd /dev/kfd /dev/dri /dev/dri/*

docker run --rm --device /dev/kfd --device /dev/dri \
  --group-add 44 --group-add 993 \
  rocm/dev-ubuntu-22.04 rocminfo

# SELinux hosts (Fedora, RHEL) only:
sudo setsebool -P container_use_devices=1
    Did it work?rocminfo inside the container lists your GPU as an agent. If it lists only the CPU agent, the devices are still not reaching it.
    bash
    docker run --rm --device /dev/kfd --device /dev/dri rocm/dev-ubuntu-22.04 rocminfo | grep -i "Marketing Name"
  6. 6

    Podman equivalents

    Podman uses CDI rather than a custom runtime. Generate a CDI specification for the GPUs once with nvidia-ctk cdi generate, then reference the device by name on each run. Regenerate the specification after a driver upgrade — a stale CDI file points at library paths that no longer exist, which produces a container that starts and then fails on the first CUDA call.

    bash
    # bash — Podman with NVIDIA
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
nvidia-ctk cdi list
podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi

# Podman with AMD
podman run --rm --device /dev/kfd --device /dev/dri \
  rocm/dev-ubuntu-22.04 rocminfo
    Did it work?The CDI device appears in nvidia-ctk cdi list and the container prints the GPU table.
    bash
    nvidia-ctk cdi list
  7. 7

    Size the model for the GPU you are passing inMost common fix

    Once the GPU reaches the container, the usual arithmetic applies and containers make it easier to get wrong: it is tempting to run several GPU services at once, and each loaded model holds its own VRAM. A container that had the whole card in testing may be sharing it with two others in Compose. Check what actually fits before you scale the stack up.

    bash
    # bash — what is holding VRAM across every container right now?
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
    Check what fits your hardwarecheck what fits before you run several GPU containers at onceOpen the VRAM checker →

If none of this worked

If the container sees the GPU but cannot reach Ollama on the host, that is a networking problem rather than a device problem and has its own page. On Windows, Docker Desktop with the WSL2 backend rides on the WSL driver plumbing, so fixing WSL fixes Docker there.

Include this when you report it
  • The exact docker run command and the full error
  • Output of nvidia-ctk --version and docker info | grep -i runtime
  • Whether Docker is rootful, rootless, or Podman
  • ls -lnd /dev/kfd /dev/dri /dev/dri/* for AMD problems

Related

Frequently asked questions

Why is the host driver not enough for a container?

Because the container needs the driver's libraries and device nodes injected into its own filesystem and namespace, which is what the NVIDIA Container Toolkit does at start. Without it Docker has no driver able to satisfy the [[gpu]] capability and refuses to start the container.

What is the difference between the two error messages?

"could not select device driver ... [[gpu]]" means the toolkit is missing or not registered. "unknown or invalid runtime name: nvidia" means it is installed but Docker's daemon.json has not been configured — you skipped nvidia-ctk runtime configure or did not restart the daemon.

Why does it work with sudo but not as my user?

You are almost certainly on rootless Docker, which reads its own daemon config from your home directory. Run nvidia-ctk runtime configure against $HOME/.config/docker/daemon.json and set no-cgroups in the container-toolkit config.

How do I pass an AMD GPU into a container?

Pass /dev/kfd and /dev/dri as devices, and add the numeric group IDs owning them with --group-add — names differ between host and image, so use the numbers from ls -lnd. On SELinux hosts also set the container_use_devices boolean.

Do I need to redo any of this after a driver upgrade?

For Docker, usually not. For Podman, yes — regenerate the CDI specification, because it records library paths that a driver upgrade changes. A stale CDI file gives you a container that starts and then fails on its first CUDA call.