Local LLM troubleshooting on Linux

On Linux the GPU either works or it is completely absent, and the absence is nearly always the kernel module rather than the card. The driver package on disk and the module in the running kernel are two different things, and anything that changes one without the other — an apt upgrade that installed a new kernel, a driver bumped while the old module was still loaded, Secure Boot refusing to load something unsigned — produces the same terse message about failing to communicate with the NVIDIA driver.

The second family of Linux problems is not the hardware at all: it is which user, and which environment, your inference is running as. The Ollama installer sets up a systemd service running as its own user, so the variables you exported in your shell configure your shell and nothing else. AMD's compute device nodes belong to the render and video groups, so ROCm works under sudo and fails as you. Containers see the GPU only if the NVIDIA Container Toolkit has been wired into the Docker runtime.

Every fix below states the Ubuntu/Debian command, and the Fedora or Arch equivalent where it genuinely differs. Start with nvidia-smi or rocminfo: if the card is not there, you are in the first group; if it is there and something still will not use it, you are in the second.