nvidia-smi stopped working after a kernel update

Linux

作者： Jakub Rusinowski · 最后更新： 2026年9月1日

AI 教育者、本地 LLM 部署工作坊讲师

The error

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

你遇到的是哪一种？

如果你看到	原因是	前往
`dkms status` lists the module against the OLD kernel only	DKMS never built the module for the kernel you just booted	Fix 2: install matching headers and rebuild
`dkms status` shows nothing at all for nvidia	The driver was installed without DKMS, so it does not follow kernels	Fix 4: reinstall the driver with DKMS support
`modprobe nvidia` says "Key was rejected by service"	Secure Boot is refusing the freshly built unsigned module	This is a different page — see Secure Boot below
The build log ends at a missing header or a compiler error	Headers absent, or the driver version predates this kernel	Fix 3: read make.log — it names the reason
You need the machine working right now	The previous kernel still has a working module	Fix 1: boot the old kernel from the GRUB menu

When you see it

Nothing about your setup changed. You ran a routine update, rebooted, and the GPU is simply not there any more — the same error you would get on a machine with no NVIDIA card installed at all.

You ran apt upgrade or dnf update, rebooted, and inference now runs on the CPU.
Unattended-upgrades installed a kernel overnight and the machine rebooted itself.
nvidia-smi fails but lspci | grep -i nvidia still lists the card, proving the hardware is fine.
You are on a distro that ships kernel updates frequently — Arch, Fedora, or Ubuntu with HWE kernels.
This has happened before and you fixed it by reinstalling the driver, which worked but took an hour.

What's actually going on

The NVIDIA kernel module is not portable between kernels: it is compiled against the exact kernel it will load into. DKMS exists to handle that — when a new kernel is installed, DKMS is meant to rebuild every registered module for it automatically. The rebuild needs the linux-headers package matching that specific kernel, and if the headers are absent the build fails. It usually fails quietly, in the middle of a long apt transaction whose output nobody reads, so the first sign of trouble is a reboot into a kernel with no NVIDIA module. lspci still lists the card because that reads the PCI bus directly; nvidia-smi fails because it talks to a driver that is not loaded. The card is fine. The module for this kernel does not exist.

How to fix it

1. Boot the previous kernel to get working again now

Before fixing anything, know that the old kernel still has a working module and is one reboot away. Hold Shift (BIOS) or press Esc (UEFI) during boot to reach the GRUB menu, choose Advanced options for …, and pick the previous kernel version. That restores the GPU immediately and takes the time pressure off. Do it if a job is running or someone is waiting on the machine — then come back and fix it properly.

bash

# bash — which kernel am I on, and which are installed?
uname -r
ls /boot/vmlinuz-*

生效了吗？ uname -r shows the older version and nvidia-smi prints a normal table again. You are running on borrowed time, not fixed.

uname -r && nvidia-smi

2. Install the matching headers and rebuild the module

This is the actual fix and it is usually three commands. dkms status tells you which kernels the module has been built for — if your running kernel is missing from that list, the headers are the likely reason. Install the headers for the running kernel specifically, not the generic metapackage, then let DKMS build everything it owes.

bash

# bash — Ubuntu / Debian
uname -r
dkms status
sudo apt install -y linux-headers-$(uname -r)
sudo dkms autoinstall
sudo modprobe nvidia

# Fedora
sudo dnf install -y kernel-devel-$(uname -r)
sudo akmods --force && sudo dracut --force

# Arch
sudo pacman -S --needed linux-headers
sudo dkms autoinstall

生效了吗？ dkms status now lists the module as installed against your running kernel, and nvidia-smi prints the GPU table. Reboot once to confirm it survives.

dkms status
nvidia-smi --query-gpu=name,driver_version --format=csv

3. Read the DKMS build log — it names the real reason

If dkms autoinstall still fails, the build log says why and almost nobody looks at it. It lives under /var/lib/dkms/nvidia/<version>/build/make.log. The last twenty lines carry the actual compiler error, and the two common ones are unambiguous: a missing header path means the headers are still not right, while an error about an unknown kernel API means the driver version is older than the kernel and no amount of rebuilding will fix it — you need a newer driver.

bash

# bash — find the log and read the end of it
ls -d /var/lib/dkms/nvidia/*/
sudo tail -n 30 /var/lib/dkms/nvidia/*/build/make.log

# Retry a single module verbosely once you have addressed the cause
sudo dkms install -m nvidia -v $(ls /var/lib/dkms/nvidia | head -1) -k $(uname -r) --verbose

生效了吗？ The verbose install ends with a "DKMS: install completed" line rather than an error, and modinfo nvidia reports a version.

modinfo nvidia | head -3

4. Reinstall the driver with DKMS if it was never registered

If dkms status prints nothing for nvidia, the driver was installed without DKMS — typically from NVIDIA's .run installer without the --dkms flag — and it will break on every kernel update from now on, not just this one. Reinstalling from your distribution's packages puts DKMS back in charge and ends the recurrence. On Ubuntu, ubuntu-drivers devices names the recommended package for your card.

⚠️ Purging driver packages removes the running module and will drop a desktop machine to a text console until the new driver is installed and loaded. Do it from a TTY, not from a terminal inside your desktop session, and have a plan for finishing without a GUI.

bash

# bash — Ubuntu: see what is recommended, then install it
ubuntu-drivers devices
sudo apt install -y nvidia-driver-580   # substitute the recommended version

# If a .run installer was used previously, remove it first:
sudo /usr/bin/nvidia-uninstall

生效了吗？ dkms status lists nvidia against your running kernel. That is the thing that makes future kernel updates uneventful.

dkms status | grep -i nvidia

5. Decide whether to hold kernel updates — and be honest about the cost

On a machine you depend on, you can pin the kernel so this cannot happen unattended. It genuinely works, and it means you stop receiving kernel security updates until you unhold and deal with the rebuild deliberately. That is a real trade rather than a free win. The middle path most people are happier with is to keep updates and simply always reboot when you have ten minutes, so a failed rebuild is discovered by you rather than by a job at 3am.

⚠️ Holding the kernel freezes security patches for it. Only do this on a machine whose exposure you understand, and set yourself a reminder to unhold and update on purpose.

bash

# bash — Ubuntu / Debian
sudo apt-mark hold linux-image-generic linux-headers-generic
apt-mark showhold

# Release it later
sudo apt-mark unhold linux-image-generic linux-headers-generic

6. Meanwhile, keep working on the CPU with a smaller model Most common fix

If the rebuild will take a while — a driver upgrade, a Secure Boot enrolment, an out-of-hours reboot — you are not stuck. A small model at Q4 is genuinely usable on a modern CPU for chat and light editing, at single-digit tokens per second rather than dozens. Knowing which size stays comfortable on your CPU is worth ten seconds now and saves the afternoon.

bash

# bash — a 1-3B model runs acceptably on CPU while the GPU is out
ollama run gemma3:1b
ollama ps   # Processor column will read 100% CPU, as expected here

Check what fits your hardware — find a model that stays usable on the CPU while the driver is out
Open the VRAM checker →

如果以上都没用

If the module now builds but still refuses to load, the kernel is rejecting it rather than missing it — that is Secure Boot, and it has its own page. If nvidia-smi fails without a kernel update having happened at all, the version mismatch page is the right one.

上报问题时请附上这些信息

Output of uname -r, dkms status and modinfo nvidia | head -3
The last 30 lines of /var/lib/dkms/nvidia/*/build/make.log
Your distribution and kernel package (apt list --installed | grep linux-image)
How the driver was installed: distro package, CUDA repo, or a .run file

Frequently asked questions

Why does a kernel update break the NVIDIA driver?

Because the driver includes a kernel module compiled against one specific kernel. A new kernel needs the module rebuilt, which DKMS normally does automatically — but the rebuild requires matching linux-headers, and it fails quietly when they are missing.

lspci still shows my GPU. Does that mean the card is fine?

Yes. lspci reads the PCI bus directly and does not need a driver, so it lists the card either way. nvidia-smi talks to the loaded kernel module, so it fails when the module is absent. The gap between those two outputs is exactly this problem.

How do I get working again immediately?

Reboot into the previous kernel from the GRUB advanced options menu. The old kernel still has a working module built for it, so the GPU comes straight back. It is a stopgap, not a fix — the new kernel still has no module.

dkms autoinstall runs but the module still will not load. What now?

Read /var/lib/dkms/nvidia/*/build/make.log. A missing header path means the headers are still wrong; an unknown-kernel-API error means the driver predates this kernel and needs upgrading. If the build succeeded and modprobe still refuses, suspect Secure Boot.

Should I stop the kernel from updating?

Only with your eyes open. apt-mark hold prevents the surprise, and also stops kernel security updates until you unhold. On most workstations the better habit is to keep updates and reboot when you have time to notice a failed rebuild yourself.