nvidia-smi stopped working after a kernel update

It worked yesterday, now the GPU is gone

Written by Jakub Rusinowski

The error

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

1. 1

   Boot the previous kernel to get working again now

   Before fixing anything, know that the old kernel still has a working module and is one reboot away. Hold Shift (BIOS) or press Esc (UEFI) during boot to reach the GRUB menu, choose Advanced options for …, and pick the previous kernel version. That restores the GPU immediately and takes the time pressure off. Do it if a job is running or someone is waiting on the machine — then come back and fix it properly.

   bashCopy

   # bash — which kernel am I on, and which are installed?
   uname -r
   ls /boot/vmlinuz-*

   Did it work?uname -r shows the older version and nvidia-smi prints a normal table again. You are running on borrowed time, not fixed.

   bashCopy

   uname -r && nvidia-smi

2. 2

   Install the matching headers and rebuild the module

   This is the actual fix and it is usually three commands. dkms status tells you which kernels the module has been built for — if your running kernel is missing from that list, the headers are the likely reason. Install the headers for the running kernel specifically, not the generic metapackage, then let DKMS build everything it owes.

   bashCopy

   # bash — Ubuntu / Debian
   uname -r
   dkms status
   sudo apt install -y linux-headers-$(uname -r)
   sudo dkms autoinstall
   sudo modprobe nvidia
   
   # Fedora
   sudo dnf install -y kernel-devel-$(uname -r)
   sudo akmods --force && sudo dracut --force
   
   # Arch
   sudo pacman -S --needed linux-headers
   sudo dkms autoinstall

   Did it work?dkms status now lists the module as installed against your running kernel, and nvidia-smi prints the GPU table. Reboot once to confirm it survives.

   bashCopy

   dkms status
   nvidia-smi --query-gpu=name,driver_version --format=csv

3. 3

   Read the DKMS build log — it names the real reason

   If dkms autoinstall still fails, the build log says why and almost nobody looks at it. It lives under /var/lib/dkms/nvidia/<version>/build/make.log. The last twenty lines carry the actual compiler error, and the two common ones are unambiguous: a missing header path means the headers are still not right, while an error about an unknown kernel API means the driver version is older than the kernel and no amount of rebuilding will fix it — you need a newer driver.

   bashCopy

   # bash — find the log and read the end of it
   ls -d /var/lib/dkms/nvidia/*/
   sudo tail -n 30 /var/lib/dkms/nvidia/*/build/make.log
   
   # Retry a single module verbosely once you have addressed the cause
   sudo dkms install -m nvidia -v $(ls /var/lib/dkms/nvidia | head -1) -k $(uname -r) --verbose

   Did it work?The verbose install ends with a "DKMS: install completed" line rather than an error, and modinfo nvidia reports a version.

   bashCopy

   modinfo nvidia | head -3

4. 4

   Reinstall the driver with DKMS if it was never registered

   If dkms status prints nothing for nvidia, the driver was installed without DKMS — typically from NVIDIA's .run installer without the --dkms flag — and it will break on every kernel update from now on, not just this one. Reinstalling from your distribution's packages puts DKMS back in charge and ends the recurrence. On Ubuntu, ubuntu-drivers devices names the recommended package for your card.

   Careful: Purging driver packages removes the running module and will drop a desktop machine to a text console until the new driver is installed and loaded. Do it from a TTY, not from a terminal inside your desktop session, and have a plan for finishing without a GUI.

   bashCopy

   # bash — Ubuntu: see what is recommended, then install it
   ubuntu-drivers devices
   sudo apt install -y nvidia-driver-580   # substitute the recommended version
   
   # If a .run installer was used previously, remove it first:
   sudo /usr/bin/nvidia-uninstall

   Did it work?dkms status lists nvidia against your running kernel. That is the thing that makes future kernel updates uneventful.

   bashCopy

   dkms status | grep -i nvidia

5. 5

   Decide whether to hold kernel updates — and be honest about the cost

   On a machine you depend on, you can pin the kernel so this cannot happen unattended. It genuinely works, and it means you stop receiving kernel security updates until you unhold and deal with the rebuild deliberately. That is a real trade rather than a free win. The middle path most people are happier with is to keep updates and simply always reboot when you have ten minutes, so a failed rebuild is discovered by you rather than by a job at 3am.

   Careful: Holding the kernel freezes security patches for it. Only do this on a machine whose exposure you understand, and set yourself a reminder to unhold and update on purpose.

   bashCopy

   # bash — Ubuntu / Debian
   sudo apt-mark hold linux-image-generic linux-headers-generic
   apt-mark showhold
   
   # Release it later
   sudo apt-mark unhold linux-image-generic linux-headers-generic

6. 6

   Meanwhile, keep working on the CPU with a smaller modelMost common fix

   If the rebuild will take a while — a driver upgrade, a Secure Boot enrolment, an out-of-hours reboot — you are not stuck. A small model at Q4 is genuinely usable on a modern CPU for chat and light editing, at single-digit tokens per second rather than dozens. Knowing which size stays comfortable on your CPU is worth ten seconds now and saves the afternoon.

   bashCopy

   # bash — a 1-3B model runs acceptably on CPU while the GPU is out
   ollama run gemma3:1b
   ollama ps   # Processor column will read 100% CPU, as expected here

   Check what fits your hardwarefind a model that stays usable on the CPU while the driver is outOpen the VRAM checker →