KernelRemoval section of DistUpgrade.cfg shouldn't require manual updating

Additionally, the Version part of KernelRemoval is expected to be a string and not a list so you can't have multiple kernel versions thereby making it so that we can only remove the release kernel or the hwe one which is also lame.

I'm wondering if we still need this functionality in u-r-u.

It offers removing removal of all autoremovable packages at the end of the upgrade for a long time and kernels are autoremoved periodically by unattended-upgrades and update-manager in Xenial and up.

IMO those are enough and since u-r-u performs autoremoval after the upgrade it is not affected by apt's LP: #1615381.

So I propose just dropping the functionality from u-r-u which would solve the maintenance problem since the kernels are removed in other ways.

Testing an upgrade from Ubuntu 18.10 to Ubuntu 19.04 I noticed that the identifyObsoleteKernels function returns a set of linux packages which was already identified as obsolete so dropping the functionality as Balint suggested seems easiest.

Relevant part of /var/log/dist-upgrade/main.log:

2019-06-19 13:31:02,029 DEBUG Obsolete: gir1.2-mutter-3 libbind9-160 libdns-export1102 libdns1102 libedataserver-1.2-23 libexempi3 libgdbm5 libhunspell-1.6-0 libicu60 libirs160 libisc-export169 libisc169 libisccc160 libisccfg160 liblouis16 liblwres160 libmutter-3-0 libntfs-3g88 liborcus-0.13-0 libperl5.26 libpoppler79 libprotobuf10 libpython3.6 libpython3.6-minimal libpython3.6-stdlib libraw16 libreadline7 libssl1.0.0 linux-headers-4.18.0-10 linux-headers-4.18.0-10-generic linux-headers-4.18.0-18 linux-headers-4.18.0-18-generic linux-image-4.18.0-10-generic linux-image-4.18.0-18-generic linux-modules-4.18.0-10-generic linux-modules-4.18.0-18-generic linux-modules-extra-4.18.0-10-generic linux-modules-extra-4.18.0-18-generic perl-modules-5.26 python-talloc python3.6 python3.6-minimal
2019-06-19 13:31:02,029 DEBUG Foreign:
2019-06-19 13:31:02,029 DEBUG identifyObsoleteKernels()
2019-06-19 13:31:04,958 DEBUG removing obsolete kernel 'linux-headers-4.18.0-10-generic'
2019-06-19 13:31:04,958 DEBUG skipping running kernel linux-headers-4.18.0-18-generic
2019-06-19 13:31:04,961 DEBUG removing obsolete kernel 'linux-image-4.18.0-10-generic'
2019-06-19 13:31:04,961 DEBUG skipping running kernel linux-image-4.18.0-18-generic
2019-06-19 13:31:04,967 DEBUG removing obsolete kernel 'linux-modules-4.18.0-10-generic'
2019-06-19 13:31:04,967 DEBUG skipping running kernel linux-modules-4.18.0-18-generic
2019-06-19 13:31:04,969 DEBUG removing obsolete kernel 'linux-modules-extra-4.18.0-10-generic'
2019-06-19 13:31:04,969 DEBUG skipping running kernel linux-modules-extra-4.18.0-18-generic
2019-06-19 13:31:06,446 DEBUG identifyObsoleteKernels found '{'linux-modules-extra-4.18.0-10-generic', 'linux-modules-4.18.0-10-generic', 'linux-image-4.18.0-10-generic', 'linux-headers-4.18.0-10-generic'}'
2019-06-19 13:31:06,447 DEBUG forced_obsoletes: ['desktop-effects', 'gnome-app-install', 'policykit-gnome', 'gnome-mount', 'linux-modules-extra-4.18.0-10-generic', 'linux-modules-4.18.0-10-generic', 'linux-image-4.18.0-10-generic', 'linux-headers-4.18.0-10-generic']

I also tested an upgrade from Ubuntu 19.04 to eoan and here we can see identifyObsoleteKernels actually attempting to do something wrong - suggesting to remove the kernel just installed. While this is because Ubuntu 19.04 and eoan currently have the same kernel version it still seems like an argument for dropping the function.

Relevant part of /var/log/dist-upgrade/main.log:

2019-06-19 14:51:33,447 DEBUG Obsolete: libisl19 libpoppler85 linux-headers-5.0.0-13 linux-headers-5.0.0-13-generic linux-image-5.0.0-13-generic linux-modules-5.0.0-13-generic linux-modules-extra-5.0.0-13-generic
2019-06-19 14:51:33,448 DEBUG Foreign:
2019-06-19 14:51:33,448 DEBUG identifyObsoleteKernels()
2019-06-19 14:51:36,381 DEBUG skipping running kernel linux-headers-5.0.0-13-generic
2019-06-19 14:51:36,381 DEBUG removing obsolete kernel 'linux-headers-5.0.0-17-generic'
2019-06-19 14:51:36,383 DEBUG skipping running kernel linux-image-5.0.0-13-generic
2019-06-19 14:51:36,383 DEBUG removing obsolete kernel 'linux-image-5.0.0-17-generic'
2019-06-19 14:51:36,387 DEBUG skipping running kernel linux-modules-5.0.0-13-generic
2019-06-19 14:51:36,387 DEBUG removing obsolete kernel 'linux-modules-5.0.0-17-generic'
2019-06-19 14:51:36,387 DEBUG skipping running kernel linux-modules-extra-5.0.0-13-generic
2019-06-19 14:51:36,388 DEBUG removing obsolete kernel 'linux-modules-extra-5.0.0-17-generic'
2019-06-19 14:51:37,872 DEBUG identifyObsoleteKernels found '{'linux-modules-extra-5.0.0-17-generic', 'linux-image-5.0.0-17-generic', 'linux-headers-5.0.0-17-generic', 'linux-modules-5.0.0-17-generic'}'
2019-06-19 14:51:37,872 DEBUG forced_obsoletes: ['desktop-effects', 'gnome-app-install', 'policykit-gnome', 'gnome-mount', 'linux-modules-extra-5.0.0-17-generic', 'linux-image-5.0.0-17-generic', 'linux-headers-5.0.0-17-generic', 'linux-modules-5.0.0-17-generic']
2019-06-19 14:51:38,179 DEBUG Unused dependencies: pppoeconf libisl19 pppconfig libpoppler85 ifupdown
2019-06-19 14:51:38,179 DEBUG remove_candidates: '{'linux-modules-extra-5.0.0-13-generic', 'pppoeconf', 'gnome-app-install', 'libisl19', 'desktop-effects', 'linux-image-5.0.0-13-generic', 'pppconfig', 'linux-headers-5.0.0-13-generic', 'linux-modules-5.0.0-17-generic', 'linux-headers-5.0.0-17-generic', 'policykit-gnome', 'ifupdown', 'linux-headers-5.0.0-13', 'gnome-mount', 'linux-image-5.0.0-17-generic', 'linux-modules-5.0.0-13-generic', 'libpoppler85', 'linux-modules-extra-5.0.0-17-generic'}'
2019-06-19 14:51:38,179 DEBUG Start checking for obsolete pkgs
2019-06-19 14:51:38,180 DEBUG skipping running kernel pkg 'linux-modules-extra-5.0.0-13-generic'
2019-06-19 14:51:38,180 DEBUG 'linux-modules-extra-5.0.0-13-generic' scheduled for remove but not safe to remove, skipping
2019-06-19 14:51:38,399 DEBUG skipping running kernel pkg 'linux-image-5.0.0-13-generic'
2019-06-19 14:51:38,399 DEBUG 'linux-image-5.0.0-13-generic' scheduled for remove but not safe to remove, skipping
2019-06-19 14:51:38,508 DEBUG skipping running kernel pkg 'linux-headers-5.0.0-13-generic'
2019-06-19 14:51:38,508 DEBUG 'linux-headers-5.0.0-13-generic' scheduled for remove but not safe to remove, skipping
2019-06-19 14:51:38,722 DEBUG package 'linux-module...

I went ahead and disabled the identifyObsoleteKernels function and here is the relevant part of /var/log/dist-upgrade/main.log:

2019-06-19 15:37:29,707 DEBUG Obsolete: libisl19 libpoppler85 linux-headers-5.0.0-13 linux-headers-5.0.0-13-generic linux-image-5.0.0-13-generic linux-modules-5.0.0-13-generic linux-modules-extra-5.0.0-13-generic
2019-06-19 15:37:29,707 DEBUG Foreign:
2019-06-19 15:37:29,707 DEBUG forced_obsoletes: ['desktop-effects', 'gnome-app-install', 'policykit-gnome', 'gnome-mount']
2019-06-19 15:37:30,001 DEBUG Unused dependencies: libisl19 pppconfig ifupdown libpoppler85 pppoeconf
2019-06-19 15:37:30,001 DEBUG remove_candidates: '{'libisl19', 'linux-image-5.0.0-13-generic', 'gnome-mount', 'policykit-gnome', 'desktop-effects', 'linux-modules-5.0.0-13-generic', 'pppconfig', 'linux-modules-extra-5.0.0-13-generic', 'gnome-app-install', 'libpoppler85', 'ifupdown', 'linux-headers-5.0.0-13', 'linux-headers-5.0.0-13-generic', 'pppoeconf'}'
2019-06-19 15:37:30,001 DEBUG Start checking for obsolete pkgs
2019-06-19 15:37:30,100 DEBUG skipping running kernel pkg 'linux-image-5.0.0-13-generic'
2019-06-19 15:37:30,100 DEBUG 'linux-image-5.0.0-13-generic' scheduled for remove but not safe to remove, skipping
2019-06-19 15:37:30,101 DEBUG skipping running kernel pkg 'linux-modules-5.0.0-13-generic'
2019-06-19 15:37:30,101 DEBUG 'linux-modules-5.0.0-13-generic' scheduled for remove but not safe to remove, skipping
2019-06-19 15:37:30,198 DEBUG skipping running kernel pkg 'linux-modules-extra-5.0.0-13-generic'
2019-06-19 15:37:30,198 DEBUG 'linux-modules-extra-5.0.0-13-generic' scheduled for remove but not safe to remove, skipping
2019-06-19 15:37:30,720 DEBUG skipping running kernel pkg 'linux-headers-5.0.0-13-generic'
2019-06-19 15:37:30,721 DEBUG 'linux-headers-5.0.0-13-generic' scheduled for remove but not safe to remove, skipping
2019-06-19 15:37:30,822 DEBUG Finish checking for obsolete pkgs
2019-06-19 15:37:30,861 DEBUG The following packages are marked for removal: linux-headers-5.0.0-13 ifupdown linux-headers-5.0.0-13-generic pppoeconf libisl19 pppconfig libpoppler85

So just removing the function seems like the best way forward.

It was suggested that testing upgrades from Ubuntu 18.04 should also be done so I tested an upgrade from Ubuntu 18.04 to Ubuntu 19.04 and eoan (which will become Ubuntu 19.10) and both of these behaved in the same way with the removal of identifyObsoleteKernels so lets do that.

This bug was fixed in the package ubuntu-release-upgrader - 1:19.10.9

---------------
ubuntu-release-upgrader (1:19.10.9) eoan; urgency=medium

  * DistUpgrade/DistUpgradeCache.py, DistUpgrade/DistUpgradeController.py:
    drop code which identified obsolete kernels and marked them for removal as
    it is redundant with functionality that exists in apt. (LP: #1828100)
  * DistUpgrade/DistUpgradeQuirks.py: Create a fallback mechanism for
    determining the version of a release (necessary for Ubuntu 18.04) when
    replacing debs with snaps. (LP: #1833753)

 -- Brian Murray <email address hidden> Fri, 21 Jun 2019 14:30:19 -0700

The case when u-r-u considered (then skipped) removing the running kernel package will be fixed with LP: #1615381 getting fixed in APT.

Until this happens it is better to keep the sanity check in tryMarkObsoleteForRemoval() and maybe extend it to cover package names ending with the kernel version without flavor, like linux-headers-4.18.0-24.

See: LP: #1828200 .