"Failed to initialize NVML: Driver/library version mismatch"

Name: LLM Configurator — GPU VRAM Checker
Author: LLM Configurator

Linux

Written by Jakub Rusinowski · Last updated September 1, 2026

Founder, LLM Configurator — AI educator & workshop leader on local LLM deployment

The error

Failed to initialize NVML: Driver/library version mismatch
NVML library version: 580.65

CUDA driver version is insufficient for CUDA runtime version

Which one is it?

If you see	The cause is	Go to
An `apt upgrade` or `dnf update` ran recently and you have not rebooted	Userspace libraries are newer than the kernel module still in memory	Fix 1: reboot — it takes 40 seconds
You cannot reboot (shared box, long job in tmux)	Same cause, but you need to swap the module in place	Fix 2: unload and reload the modules
`rmmod` says the module is in use	A display manager, X/Wayland, or a process still holds the GPU	Fix 3: find what is holding it
It appeared after suspend/resume rather than an upgrade	GPU discovery failed on resume; the UVM module needs reloading	Fix 4: reload nvidia_uvm
`modinfo` and `nvidia-smi` report the same version and it still fails	Two driver installs are fighting — distro packages plus a .run file	Fix 5: settle on one install method

When you see it

This one alarms people because nothing obviously happened. There was no kernel update, no reboot, no configuration change — and nvidia-smi has gone from working to a version-mismatch error mid-session.

A package manager upgraded the NVIDIA driver in the background, possibly as part of an unrelated update.
Unattended-upgrades ran while you were working.
You installed CUDA or a new driver an hour ago and carried on without rebooting.
The machine came back from suspend and the GPU stopped being visible.
You have both distribution driver packages and a .run installer on the same machine.

What's actually going on

The NVIDIA driver is two halves that must match: a kernel module loaded into the running kernel, and userspace libraries on disk that talk to it. A package upgrade replaces the libraries immediately, but it cannot replace a module that is currently loaded and in use — so the old module stays in memory while the new libraries sit on disk. nvidia-smi links against the new NVML library, asks the old module its version, and refuses to continue. Nothing is corrupt and nothing is lost. The running kernel is simply holding a stale module, which is why a reboot fixes it completely and why the manual fix is nothing more than swapping the module out. The CUDA-side phrasing — "CUDA driver version is insufficient for CUDA runtime version" — is the same fault seen from a different library.

How to fix it

1. Reboot

The honest first answer. A reboot loads the new module against the new libraries and takes about forty seconds, and it is what you would do anyway if the manual route below goes wrong. Everything else on this page exists for machines you genuinely cannot restart — a shared box, a training run you refuse to lose, a server with other users on it. If none of those apply to you, stop reading and reboot.

bash

# bash
sudo reboot

Did it work? After the reboot, nvidia-smi prints a normal table and the reported driver version matches what your package manager installed.

nvidia-smi --query-gpu=driver_version --format=csv
modinfo nvidia | grep ^version

2. Unload and reload the modules in dependency order

Without a reboot, you need to remove the stale module and load the new one. Order matters: nvidia is held by nvidia_uvm, nvidia_drm and nvidia_modeset, so they come out first and the base module last. On a desktop this will drop you to a text console, because the display stack is one of the things holding the module — plan to run it from a TTY (Ctrl+Alt+F3), not from a terminal inside the session you are about to kill.

⚠️ On a machine with a graphical session this terminates your desktop. Save your work first. On a headless server it is safe and non-disruptive to anything that is not using the GPU.

bash

# bash — as root, ideally from a TTY on a desktop machine
sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia
sudo modprobe nvidia
sudo modprobe nvidia_uvm

nvidia-smi

Did it work? nvidia-smi works again, and modinfo nvidia now reports the same version as the userspace libraries.

modinfo nvidia | grep ^version
nvidia-smi --query-gpu=driver_version --format=csv,noheader

3. Find whatever is still holding the GPU

If rmmod reports the module is in use, something has the device open. fuser -v /dev/nvidia* names the processes; lsof /dev/nvidia* is the alternative where fuser is not installed. The usual holders are your inference server, a Python process that has not exited, a container, and — on a desktop — the display manager. Stop the ones you can, and accept that on a desktop the display stack is not one of them, which is what fix 2's TTY note is about.

bash

# bash
sudo fuser -v /dev/nvidia*
sudo lsof /dev/nvidia* 2>/dev/null | head -20

# Common holders, stopped politely
sudo systemctl stop ollama
docker ps -q | xargs -r docker stop

# Desktop only: stop the display manager (this ends your GUI session)
sudo systemctl stop gdm3     # or lightdm, sddm

Did it work? sudo fuser -v /dev/nvidia* prints nothing. Only then will rmmod succeed.

sudo fuser -v /dev/nvidia*

4. After suspend/resume, reload the UVM module specifically

There is a documented variant of this that has nothing to do with upgrades: after a suspend and resume, GPU discovery can fail with the same class of symptom. Ollama's own troubleshooting documentation gives the workaround as reloading the Unified Memory module on its own, which is much less disruptive than the full unload in fix 2 — it does not touch the display stack. nvidia-modprobe -u is the documented alternative for loading the UVM driver.

bash

# bash — the documented suspend/resume workaround
sudo rmmod nvidia_uvm
sudo modprobe nvidia_uvm

# Or, equivalently
sudo nvidia-modprobe -u

sudo systemctl restart ollama

Did it work? The module is loaded and your inference server sees the GPU again.

lsmod | grep nvidia_uvm
ollama ps

5. Settle on one driver installation method

If this recurs, or if modinfo and nvidia-smi agree on the version and it still fails, you probably have two driver installs on the same machine — distribution packages and an NVIDIA .run installer, each convinced it owns /usr/lib/x86_64-linux-gnu/libnvidia-*. Pick one. Distribution packages are the right default for almost everyone because they participate in DKMS and in upgrades; the .run installer is for cases where you specifically need a version your distro does not carry.

⚠️ Removing a driver install leaves the machine without a working GPU driver until the replacement is in place. Do this from a TTY with the replacement packages already downloaded, not over SSH on a machine you cannot reach physically.

bash

# bash — remove a previous .run install, then use distro packages
sudo /usr/bin/nvidia-uninstall
sudo apt install --reinstall -y nvidia-driver-580   # your recommended version
sudo reboot

6. Understand the NVML error codes you may see alongside this Most common fix

NVML surfaces a small set of numbers that narrow the problem quickly: 3 (not initialized) and 999 (unknown) usually accompany this mismatch; 46 (device unavailable) points at a GPU in a bad state, often after a fallen-over job; 100 (no device) means no GPU was found at all, which is the DKMS or Secure Boot story rather than this one. Reading the code first saves you trying this page's fixes on a problem that belongs to another.

bash

# bash — the full error text, and what is loaded right now
nvidia-smi 2>&1 | head -5
lsmod | grep -E "^nvidia"
cat /proc/driver/nvidia/version 2>/dev/null

Check what fits your hardware — confirm what your card runs once the driver is talking to it again
Open the VRAM checker →

If none of this worked

If the versions now match and the GPU still is not usable inside a container, the container runtime is the next layer down. If nvidia-smi fails with a communication error rather than a version mismatch, the module is missing rather than stale, and that is the kernel-update page.

Include this when you report it

The full first five lines of nvidia-smi output
modinfo nvidia | grep ^version and cat /proc/driver/nvidia/version
Output of lsmod | grep nvidia and sudo fuser -v /dev/nvidia*
Whether the driver came from distro packages, the CUDA repo, or a .run installer

A model that fits most setups:

View model & requirements →

Frequently asked questions

What does "Driver/library version mismatch" actually mean?

The NVIDIA userspace libraries on disk were upgraded while the old kernel module was still loaded, so the two halves of the driver no longer agree on a version. It is a stale module in memory, not corruption, and a reboot resolves it completely.

Can I fix it without rebooting?

Yes. Stop everything holding the GPU, then sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia and sudo modprobe nvidia. On a desktop this ends your graphical session because the display stack holds the module, so run it from a TTY.

Why does rmmod say the module is in use?

Something still has a GPU device node open — an inference server, a Python process, a container, or the display manager. sudo fuser -v /dev/nvidia* names them. The module cannot be removed until that list is empty.

My GPU disappeared after suspend rather than an upgrade. Same fix?

Related but narrower. Ollama documents reloading just the Unified Memory module — sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm, or sudo nvidia-modprobe -u — which avoids disturbing the display stack. Restart your inference server afterwards.

What do the NVML error numbers mean?

Code 3 is "not initialized" and 999 is "unknown", both typical of this mismatch. Code 46 is "device unavailable", usually a GPU left in a bad state. Code 100 is "no device", which means the module is missing entirely rather than stale.