"Failed to initialize NVML: Driver/library version mismatch"

nvidia-smi errors but I never rebooted

Written by Jakub Rusinowski

The error

Failed to initialize NVML: Driver/library version mismatch
NVML library version: 580.65

CUDA driver version is insufficient for CUDA runtime version

1. 1

   Reboot

   The honest first answer. A reboot loads the new module against the new libraries and takes about forty seconds, and it is what you would do anyway if the manual route below goes wrong. Everything else on this page exists for machines you genuinely cannot restart — a shared box, a training run you refuse to lose, a server with other users on it. If none of those apply to you, stop reading and reboot.

   bashCopy

   # bash
   sudo reboot

   Did it work?After the reboot, nvidia-smi prints a normal table and the reported driver version matches what your package manager installed.

   bashCopy

   nvidia-smi --query-gpu=driver_version --format=csv
   modinfo nvidia | grep ^version

2. 2

   Unload and reload the modules in dependency order

   Without a reboot, you need to remove the stale module and load the new one. Order matters: nvidia is held by nvidia_uvm, nvidia_drm and nvidia_modeset, so they come out first and the base module last. On a desktop this will drop you to a text console, because the display stack is one of the things holding the module — plan to run it from a TTY (Ctrl+Alt+F3), not from a terminal inside the session you are about to kill.

   Careful: On a machine with a graphical session this terminates your desktop. Save your work first. On a headless server it is safe and non-disruptive to anything that is not using the GPU.

   bashCopy

   # bash — as root, ideally from a TTY on a desktop machine
   sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia
   sudo modprobe nvidia
   sudo modprobe nvidia_uvm
   
   nvidia-smi

   Did it work?nvidia-smi works again, and modinfo nvidia now reports the same version as the userspace libraries.

   bashCopy

   modinfo nvidia | grep ^version
   nvidia-smi --query-gpu=driver_version --format=csv,noheader

3. 3

   Find whatever is still holding the GPU

   If rmmod reports the module is in use, something has the device open. fuser -v /dev/nvidia* names the processes; lsof /dev/nvidia* is the alternative where fuser is not installed. The usual holders are your inference server, a Python process that has not exited, a container, and — on a desktop — the display manager. Stop the ones you can, and accept that on a desktop the display stack is not one of them, which is what fix 2's TTY note is about.

   bashCopy

   # bash
   sudo fuser -v /dev/nvidia*
   sudo lsof /dev/nvidia* 2>/dev/null | head -20
   
   # Common holders, stopped politely
   sudo systemctl stop ollama
   docker ps -q | xargs -r docker stop
   
   # Desktop only: stop the display manager (this ends your GUI session)
   sudo systemctl stop gdm3     # or lightdm, sddm

   Did it work?sudo fuser -v /dev/nvidia* prints nothing. Only then will rmmod succeed.

   bashCopy

   sudo fuser -v /dev/nvidia*

4. 4

   After suspend/resume, reload the UVM module specifically

   There is a documented variant of this that has nothing to do with upgrades: after a suspend and resume, GPU discovery can fail with the same class of symptom. Ollama's own troubleshooting documentation gives the workaround as reloading the Unified Memory module on its own, which is much less disruptive than the full unload in fix 2 — it does not touch the display stack. nvidia-modprobe -u is the documented alternative for loading the UVM driver.

   bashCopy

   # bash — the documented suspend/resume workaround
   sudo rmmod nvidia_uvm
   sudo modprobe nvidia_uvm
   
   # Or, equivalently
   sudo nvidia-modprobe -u
   
   sudo systemctl restart ollama

   Did it work?The module is loaded and your inference server sees the GPU again.

   bashCopy

   lsmod | grep nvidia_uvm
   ollama ps

5. 5

   Settle on one driver installation method

   If this recurs, or if modinfo and nvidia-smi agree on the version and it still fails, you probably have two driver installs on the same machine — distribution packages and an NVIDIA .run installer, each convinced it owns /usr/lib/x86_64-linux-gnu/libnvidia-*. Pick one. Distribution packages are the right default for almost everyone because they participate in DKMS and in upgrades; the .run installer is for cases where you specifically need a version your distro does not carry.

   Careful: Removing a driver install leaves the machine without a working GPU driver until the replacement is in place. Do this from a TTY with the replacement packages already downloaded, not over SSH on a machine you cannot reach physically.

   bashCopy

   # bash — remove a previous .run install, then use distro packages
   sudo /usr/bin/nvidia-uninstall
   sudo apt install --reinstall -y nvidia-driver-580   # your recommended version
   sudo reboot

6. 6

   Understand the NVML error codes you may see alongside thisMost common fix

   NVML surfaces a small set of numbers that narrow the problem quickly: 3 (not initialized) and 999 (unknown) usually accompany this mismatch; 46 (device unavailable) points at a GPU in a bad state, often after a fallen-over job; 100 (no device) means no GPU was found at all, which is the DKMS or Secure Boot story rather than this one. Reading the code first saves you trying this page's fixes on a problem that belongs to another.

   bashCopy

   # bash — the full error text, and what is loaded right now
   nvidia-smi 2>&1 | head -5
   lsmod | grep -E "^nvidia"
   cat /proc/driver/nvidia/version 2>/dev/null

   Check what fits your hardwareconfirm what your card runs once the driver is talking to it againOpen the VRAM checker →