Disabling unattended upgrades of CUDA toolkit

CUDA Driver/library mismatch

Per default Ubuntu runs automatic upgrades on APT installed packages every night, a process called unattended upgrades. This will sometimes cause issues when the unattended upgrade process tries to update the CUDA toolkit, as it will be able to update the libraries, but cannot update the driver, as it is in use by some application. The result is that the CUDA driver cannot be used because of the driver-library mismatch and the Agile Live applications will fail to start. In case an Agile Live application fails to start with some CUDA error, the easiest way to see if this is because of a driver-library mismatch, is to run nvidia-smi in a terminal and check the output:

$ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch

In case this happens, just reboot the machine and the new CUDA driver should be loaded instead of the old one at startup. (It is sometimes possible to manually reload the CUDA driver without rebooting, but this can be troublesome as you have to find all applications using the driver)

How to disable unattended upgrades of CUDA toolkit

One way to get rid of these unpleasant surprises when the applications suddenly no longer can start, without disabling unattended upgrades for other packages, is to blacklist the CUDA toolkit from the unattended upgrades process. This will make Ubuntu still update other packages, but CUDA toolkit will require a manual upgrade, which can be done at a suitable time point.

To blacklist the CUDA driver, open the file /etc/apt/apt.conf.d/50unattended-upgrades in a text editor and look for the Unattended-Upgrade::Package-Blacklist block. Add "cuda" and "(lib)?nvidia" to the list, like so:

// Python regular expressions, matching packages to exclude from upgrading
Unattended-Upgrade::Package-Blacklist {
    "cuda";
    "(lib)?nvidia";
};

This will disable unattended upgrades for packages with names starting with cuda, libnvidia or just nvidia.

How to manually update CUDA toolkit

Once you want to manually update the CUDA toolkit and the driver, run the following commands:

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install cuda

This will likely trigger the driver-library mismatch in case you run nvidia-smi. Then reboot the machine to reload the new driver.

Sometimes a reboot is required before the cuda package can be installed, so in case the upgrade did not work the first time, try a reboot, then apt install cuda and then reboot again.