nvidia-375 DKMS module not recompiled on upgrade to 17.04

Status changed to 'Confirmed' because the bug affects multiple users.

Attached are the logs

Here are updated logs without system76-driver-nvidia. The same issue occurs.

From apt-term.log we can see:

Setting up nvidia-375 (375.39-0ubuntu5) ...^M
/sbin/ldconfig.real: /usr/lib/nvidia-375/libEGL.so.1 is not a symbolic link^M
^M
/sbin/ldconfig.real: /usr/lib32/nvidia-375/libEGL.so.1 is not a symbolic link^M
^M
update-initramfs: deferring update (trigger activated)^M
update-initramfs: Generating /boot/initrd.img-4.8.0-46-generic^M
INFO:Enable nvidia-375^M
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/lenovo_thinkpad^M
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/put_your_quirks_here^M
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/dell_latitude^M
Loading new nvidia-375-375.39 DKMS files...^M
Building for 4.8.0-46-generic^M
Building for architecture x86_64^M
Building initial module for 4.8.0-46-generic^M
Done.^M
^M
nvidia_375:^M
Running module version sanity check.^M
 - Original module^M
   - No original module exists within this kernel^M
 - Installation^M
   - Installing to /lib/modules/4.8.0-46-generic/updates/dkms/^M
^M
nvidia_375_modeset.ko:^M
Running module version sanity check.^M
 - Original module^M
   - No original module exists within this kernel^M
 - Installation^M
   - Installing to /lib/modules/4.8.0-46-generic/updates/dkms/^M
^M
nvidia_375_drm.ko:^M
Running module version sanity check.^M
 - Original module^M
   - No original module exists within this kernel^M
 - Installation^M
   - Installing to /lib/modules/4.8.0-46-generic/updates/dkms/^M
^M
nvidia_375_uvm.ko:^M
Running module version sanity check.^M
 - Original module^M
   - No original module exists within this kernel^M
 - Installation^M
   - Installing to /lib/modules/4.8.0-46-generic/updates/dkms/^M
^M
depmod...^M
^M
DKMS: install completed.^M

So the DKMS module is built it's just built for the old (Yakkety) kernel.

I think this happens with other DKMS packages, like `virtualbox-dkms`

I don't think virtualbox has this issue

Is it possible to get the logs from when you do run `apt install --reinstall nvidia-375` ? I'm trying to figure out what is missing from the logs of the upgrade / why it behaves differently on reinstall.

Additionally, do you reboot then reinstall or reinstall and then reboot? The description seems to say both things.

I just did an upgrade from Y to Z with the virtualbox packages installed and observed the same behavior as with the nvidia DKMS modules.

Setting up virtualbox-dkms (5.1.18-dfsg-1build1) ...^M
Loading new virtualbox-5.1.18 DKMS files...^M
Building for 4.8.0-46-generic^M
Building initial module for 4.8.0-46-generic^M
Done.^M
^M
vboxdrv:^M
Running module version sanity check.^M
 - Original module^M
   - No original module exists within this kernel^M
 - Installation^M
   - Installing to /lib/modules/4.8.0-46-generic/updates/dkms/^M
^M
vboxnetadp.ko:^M
Running module version sanity check.^M
 - Original module^M
   - No original module exists within this kernel^M
 - Installation^M
   - Installing to /lib/modules/4.8.0-46-generic/updates/dkms/^M
^M
vboxnetflt.ko:^M
Running module version sanity check.^M
 - Original module^M
   - No original module exists within this kernel^M
 - Installation^M
   - Installing to /lib/modules/4.8.0-46-generic/updates/dkms/^M
^M
vboxpci.ko:^M
Running module version sanity check.^M
 - Original module^M
   - No original module exists within this kernel^M
 - Installation^M
   - Installing to /lib/modules/4.8.0-46-generic/updates/dkms/^M
^M
depmod...^M
^M
DKMS: install completed.^M

The modules were only built for the 4.8 kernel.

bdmurray@clean-yakkety-amd64:~$ ls /lib/modules/4.10.0-19-generic/
build kernel modules.alias.bin modules.builtin.bin modules.dep.bin modules.order modules.symbols vdso
initrd modules.alias modules.builtin modules.dep modules.devname modules.softdep modules.symbols.bin
bdmurray@clean-yakkety-amd64:~$ ls /lib/modules/4.8.0-46-generic/updates/dkms/
vboxdrv.ko vboxnetadp.ko vboxnetflt.ko vboxpci.ko

Reinstall virtualbox-dkms after a reboot causes the modules to be built for the new / running kernel.

@Brian without looking I would wild guess a general dkms issue

Part of the problem may be that linux-headers is installed after linux-image so there are no headers when the dkms kernel postinst script is run. From /var/log/dist-upgrade/apt-term.log:

3644 Setting up linux-image-4.10.0-19-generic (4.10.0-19.21) ...^M
3645 Running depmod.^M
3646 update-initramfs: deferring update (hook will be called later)^M
3647 Examining /etc/kernel/postinst.d.^M
3648 run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic^M
3649 run-parts: executing /etc/kernel/postinst.d/dkms 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic^M
3650 run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic^M
3651 update-initramfs: Generating /boot/initrd.img-4.10.0-19-generic^M
3652 run-parts: executing /etc/kernel/postinst.d/pm-utils 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic^M
3653 run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic^M
3654 run-parts: executing /etc/kernel/postinst.d/update-notifier 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic^M
3655 run-parts: executing /etc/kernel/postinst.d/zz-update-grub 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic^M
3656 Generating grub configuration file ...^M
3657 Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.^M
3658 Found linux image: /boot/vmlinuz-4.10.0-19-generic^M
3659 Found initrd image: /boot/initrd.img-4.10.0-19-generic^M
3660 Found linux image: /boot/vmlinuz-4.8.0-26-generic^M
3661 Found initrd image: /boot/initrd.img-4.8.0-26-generic^M
3662 Found linux image: /boot/vmlinuz-4.8.0-22-generic^M
3663 Found initrd image: /boot/initrd.img-4.8.0-22-generic^M
3664 Found memtest86+ image: /boot/memtest86+.elf^M
3665 Found memtest86+ image: /boot/memtest86+.bin^M
3666 done^M
3667 Processing triggers for doc-base (0.10.7) ...^M
3668 Processing 16 changed doc-base files, 1 added doc-base file...^M
3669 Setting up telnet (0.17-41) ...^M
3670 Setting up account-plugin-flickr (0.13+17.04.20170314-0ubuntu1) ...^M
3671 Setting up zeitgeist-core (1.0-0ubuntu2) ...^M
3672 Setting up indicator-transfer (0.2+17.04.20170215-0ubuntu2) ...^M
3673 Removing obsolete conffile /etc/xdg/autostart/indicator-transfer.desktop ...^M
3674 Setting up guile-2.0-libs:amd64 (2.0.13+1-4) ...^M
3675 Setting up libpango-1.0-0:amd64 (1.40.4-1) ...^M
3676 Setting up libproxy1-plugin-networkmanager:amd64 (0.4.14-2) ...^M
3677 Setting up libboost-filesystem1.62.0:amd64 (1.62.0+dfsg-4) ...^M
3678 Setting up libhtml-parser-perl (3.72-3) ...^M
3679 Setting up libpython2.7:amd64 (2.7.13-2) ...^M
3680 Setting up libqt5sql5:amd64 (5.7.1+dfsg-2ubuntu4~1) ...^M
3681 Setting up ftp (0.17-34) ...^M
3682 Setting up libcgi-pm-perl (4.35-1) ...^M
3683 Setting up libboost-log1.61.0 (1.61.0+dfsg-3build1) ...^M
3684 Setting up upstart (1.13.2-0ubuntu35) ...^M
3685 Installing new version of config file /etc/upstart-xsessions ...^M
3686 Setting up linux-headers-4.10.0-19-generic (4.10.0-19.21) ...^M
3687 Examining /etc/kernel/header_postinst.d.^M
3688 run-parts: executing /etc/kernel/header_postinst.d/dkms...

On Wed, Apr 12, 2017 at 12:49 PM, Brian Murray <email address hidden> wrote:
> Part of the problem may be that linux-headers is installed after linux-
> image so there are no headers when the dkms kernel postinst script is
> run. From /var/log/dist-upgrade/apt-term.log:

But shouldn't the last line:

> 3644 Setting up linux-image-4.10.0-19-generic (4.10.0-19.21) ...^M
...
> 3686 Setting up linux-headers-4.10.0-19-generic (4.10.0-19.21) ...^M
> 3687 Examining /etc/kernel/header_postinst.d.^M
> 3688 run-parts: executing /etc/kernel/header_postinst.d/dkms 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic^M

have triggered the rebuild? It seems like either dkms or this postinst
is not doing the right thing?

Are you able to run it manually?

Yes, manually running it causes the modules to be built for the new kernel.

I performed another upgrade without ubuntu-release-upgrader (edit sources, apt-get update, apt-get dist-upgrade) and the modules were not built for the new kernel.

bdmurray@clean-yakkety-amd64:~$ dkms status
virtualbox, 5.1.18, 4.8.0-26-generic, x86_64: installed

Reinstalling linux-headers-4.10.0-19-generic again resolved the issue.

With the attached patch the modules are compiled for the right version for me. It seems like the postinst wasn't working if the passed in version (from the kernel's postinst.d) happened to be the newest kernel. I'm not sure why that stopped working. It could be that this code is meant to return nothing in the newest-kernel case, and some other part of the code is meant to add the passed-in version to the list of kernels to build for. I'm not sure.

Please could someone review? If you're comfortable, you can upload otherwise I can try to do that tomorrow along with forwarding the patch.

The bug needs SRUifying too.

The function says above it

"# Get the most recent kernel on Debian based systems. This keeps
# into account both the version and the ABI. If the current kernel
# is the most recent kernel then the function will print a null string."

so I think that my previous patch is wrong.

Also this appears to not reproduce every time :(

ubuntu@autopkgtest:/lib/modules$ find -name wl.ko
./4.10.0-19-generic/updates/dkms/wl.ko
./4.8.0-46-generic/updates/dkms/wl.ko

Here's my take 2, which was forwarded upstream. Could someone review/test please?

With my patch, tested like so

  - boot a yakkety cloud vm
  - install bcmwl-kernel-source virtualbox-dkms ubuntu-release-upgrader-core from yakkety
  - install dkms from comment #19
  - do-release-upgrade -d
  - [...]
  - dkms status

ubuntu@autopkgtest:~$ sudo dkms status
bcmwl, 6.30.223.271+bdcom, 4.10.0-19-generic, x86_64: installed
bcmwl, 6.30.223.271+bdcom, 4.8.0-46-generic, x86_64: installed
virtualbox, 5.1.18, 4.10.0-19-generic, x86_64: installed
virtualbox, 5.1.18, 4.8.0-46-generic, x86_64: installed

But I would like someone that's not me to test please.

Status changed to 'Confirmed' because the bug affects multiple users.

I tested Laney's patch via the following:

- boot a yakkety desktop virtual machine
- installed virtualbox-dkms
- modified /etc/apt/sources.list for zesty
- installed new dkms package
- edited /usr/lib/dkms/common.postinst
- modified /etc/apt/sources.list for yakkety
- ran do-release-upgrade -d
- [...]
- dkms status

bdmurray@clean-yakkety-amd64:~$ dkms status
virtualbox, 5.1.18, 4.10.0-19-generic, x86_64: installed
virtualbox, 5.1.18, 4.8.0-26-generic, x86_64: installed

Also in /var/log/dist-upgrade/apt.log we can see:

Setting up virtualbox-dkms (5.1.18-dfsg-1build1) ...^M
Loading new virtualbox-5.1.18 DKMS files...^M
Building for 4.8.0-26-generic 4.10.0-19-generic^M
Building initial module for 4.8.0-26-generic^M

Notice it is building for two kernels now not one.

Hello Jeremy, or anyone else affected,

Accepted dkms into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.3-3ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Ok, proceeded with the verification steps as per comments #20 and #23. When having two kernels installed after upgrade I see:

ubuntu@yakkety-test:~$ dkms status
bcmwl, 6.30.223.271+bdcom, 4.10.0-19-generic, x86_64: installed
bcmwl, 6.30.223.271+bdcom, 4.8.0-45-generic, x86_64: installed
virtualbox, 5.1.18, 4.10.0-19-generic, x86_64: installed
virtualbox, 5.1.18, 4.8.0-45-generic, x86_64: installed

I installed dkms from zesty-proposed and then upgraded from Y to Z.

bdmurray@clean-yakkety-amd64:~$ do-release-upgrade -d
Checking for a new Ubuntu release
Get:1 Upgrade tool signature [836 B]
Get:2 Upgrade tool [1,268 kB]
Fetched 1,269 kB in 0s (0 B/s)
authenticate 'zesty.tar.gz' against 'zesty.tar.gz.gpg'
extracting 'zesty.tar.gz'
[screen is terminating]
bdmurray@clean-yakkety-amd64:~$ dkms status
virtualbox, 5.1.18, 4.10.0-19-generic, x86_64: installed
virtualbox, 5.1.18, 4.8.0-46-generic, x86_64: installed
bdmurray@clean-yakkety-amd64:~$ grep "Building for" /var/log/dist-upgrade/apt-term.log
Building for 4.8.0-46-generic 4.10.0-19-generic
bdmurray@clean-yakkety-amd64:~$ apt-cache policy dkms
dkms:
  Installed: 2.3-3ubuntu1
  Candidate: 2.3-3ubuntu1
  Version table:
 *** 2.3-3ubuntu1 100
        100 /var/lib/dpkg/status
     2.3-3 500
        500 http://192.168.10.7/ubuntu zesty/main amd64 Packages
        500 http://192.168.10.7/ubuntu zesty/main i386 Packages

Also, after using the dkms from proposed, after update I see:

ubuntu@yakkety2:~$ grep "Building for" /var/log/dist-upgrade/apt-term.log
Building for 4.8.0-45-generic 4.10.0-19-generic
Building for 4.8.0-45-generic 4.10.0-19-generic
Building for architecture x86_64

This bug was fixed in the package dkms - 2.3-3ubuntu1

---------------
dkms (2.3-3ubuntu1) zesty; urgency=medium

  * 0013-postinst-Fix-newest-kernel-detection.patch: dkms_common.postinst: We
    were passing a list of kernels to get_newest_kernel(), when it expected a
    single kernel as its first argument - the currently running kernel. This
    in turn meant that the sometimes (if the first thing in the list was
    itself the newest kernel), the wrong kernel was passed into
    _get_newest_kernel_debian(). As a result this falsely reported that there
    was no 'newest' kernel available and then didn't build modules for it.
    Drop the parameter to get_newest_kernel(), and instead rely on
    CURRENT_KERNEL being set to $(uname -r). (LP: #1681566)

 -- Iain Lane <email address hidden> Thu, 13 Apr 2017 15:50:18 +0100

The verification of the Stable Release Update for dkms has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Shouldn't we also fix yakkety and company?
Are we sure upgrade will use the new dkms?

(probably yes, because all postinst are done after all package upgrade, right?)

Sweet, thanks adconrad!

If you have a system currently running yakkety or that upgraded to you yakkety you could look to see how many kernels the modules were built for e.g. here's a yakkety system of mine:

[ 9:05AM 10517 ] [ bdmurray@impulse:/tmp ]
 $ grep "Building for" /var/log/dist-upgrade/apt-term.log
Building for 4.4.0-66-generic and 4.8.0-41-generic
Building for 4.4.0-66-generic and 4.8.0-41-generic
Building for architecture x86_64
Building for 4.4.0-66-generic and 4.8.0-41-generic

I'm sure there are plenty of apt-term.log files attached to upgrade bug reports in LP one could also check.

I found some of those attachments I mentioned:

$ for log in $(find . -name VarLogDistupgradeApttermlog.txt); do grep -H "Building for.*4.8" $log; done
./bug-1646551/VarLogDistupgradeApttermlog.txt:Building for 4.4.0-47-generic and 4.8.0-27-generic
./bug-1646551/VarLogDistupgradeApttermlog.txt:Building for 4.4.0-47-generic and 4.8.0-27-generic
./bug-1646551/VarLogDistupgradeApttermlog.txt:Building for 4.4.0-47-generic and 4.8.0-27-generic
./bug-1646551/VarLogDistupgradeApttermlog.txt:Building for 4.4.0-47-generic and 4.8.0-27-generic
./bug-1646551/VarLogDistupgradeApttermlog.txt:Building for 4.4.0-47-generic and 4.8.0-27-generic
./bug-1637466/VarLogDistupgradeApttermlog.txt:Building for 4.4.0-43-generic and 4.8.0-22-generic
./bug-1670785/VarLogDistupgradeApttermlog.txt:Building for 4.4.0-38-generic and 4.8.0-17-generic
./bug-1648952/VarLogDistupgradeApttermlog.txt:Building for 4.4.0-47-generic and 4.8.0-27-generic

Still happens with dkms modules in general: https://bugs.launchpad.net/bugs/830915

We still have this issue for our custom drivers.
What about this patch? https://github.com/dell/dkms/issues/17 I think it has not been applied to Ubunutu's version.