ubuntu-release-upgrader crashed with KeyError: "The cache has no package named 'ubuntu-desktop'"

Here's the Traceback:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 194, in __getitem__
    return self._weakref[key]
  File "/usr/lib/python3.5/weakref.py", line 131, in __getitem__
    o = self.data[key]()
KeyError: 'ubuntu-desktop'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 198, in __getitem__
    rawpkg = self._cache[key]
KeyError: 'ubuntu-desktop'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-wtsb6vse/bionic", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-wtsb6vse/DistUpgrade/DistUpgradeMain.py", line 238, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-wtsb6vse/DistUpgrade/DistUpgradeController.py", line 2017, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-wtsb6vse/DistUpgrade/DistUpgradeController.py", line 1863, in fullUpgrade
    if not self.doPostInitialUpdate():
  File "/tmp/ubuntu-release-upgrader-wtsb6vse/DistUpgrade/DistUpgradeController.py", line 894, in doPostInitialUpdate
    self.quirks.run("PostInitialUpdate")
  File "/tmp/ubuntu-release-upgrader-wtsb6vse/DistUpgrade/DistUpgradeQuirks.py", line 113, in run
    func()
  File "/tmp/ubuntu-release-upgrader-wtsb6vse/DistUpgrade/DistUpgradeQuirks.py", line 132, in bionicPostInitialUpdate
    if self.controller.cache['ubuntu-desktop'].is_installed and \
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 200, in __getitem__
    raise KeyError('The cache has no package named %r' % key)
KeyError: "The cache has no package named 'ubuntu-desktop'"

It's seems strange that 'ubuntu-desktop' wouldn't exist in the cache but there you go.

Status changed to 'Confirmed' because the bug affects multiple users.

The package 'ubuntu-desktop' doesn't exist for 16.04 on s390x where I encountered this error.
Adding a bionic repository to sources.list without upgrading works as a hacky workaround since this package exists in bionic and the failing check is simply a verification if this package is installed.

Similar to Frederik, I had to much with sources.list, adding a reference to Zesty.

In my case it was a Pine64 running:

Linux string.rvu.io 3.10.105-bsp-1.2-ayufan-99 #1 SMP PREEMPT Fri Aug 4 08:36:39 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

Upgrading from:

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="17.10 (Artful Aardvark)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.10"
VERSION_ID="17.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=artful
UBUNTU_CODENAME=artful

Ultimately the following sources.list is what worked:

$ cat /etc/apt/sources.list
## EOL upgrade sources.list
# Required
deb http://old-releases.ubuntu.com/ubuntu/ zesty main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ artful main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ artful-updates main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ artful-security main restricted universe multiverse

# Optional
#deb http://old-releases.ubuntu.com/ubuntu/ artful-backports main restricted universe multiverse

Strike that. Ultimately it didn't *really* resolve the issue for me because there are no aarch64 packages on the referenced Bionic repos:

Error during update

A problem occurred during the update. This is usually some sort of
network problem, please check your network connection and retry.

E:Failed to fetch
http://us.archive.ubuntu.com/ubuntu/dists/bionic/main/binary-arm64/Packages
404 Not Found [IP: 2001:67c:1562::19 80], E:Failed to fetch
http://us.archive.ubuntu.com/ubuntu/dists/bionic-updates/main/binary-arm64/Packages
404 Not Found [IP: 2001:67c:1562::19 80], E:Failed to fetch
http://us.archive.ubuntu.com/ubuntu/dists/bionic-security/main/binary-arm64/Packages
404 Not Found [IP: 2001:67c:1562::19 80], E:Some index files failed
to download. They have been ignored, or old ones used instead.

Not limited to s390x, but affecting it, too - hence adding z project.

The error reports in the Ubuntu Error Tracker are from i386 and amd64 systems so the issue is not specific to s390x.

However, as Frank points out there is a s390x-specific issue here in that the ubuntu-desktop package does not exist at all on s390x but does exist on other architectures.

So this check will currently ALWAYS fail on s390x.

(One wonders why ubuntu-desktop exists on s390x though given that this was deliberately not shipped on s390x in xenial.)

This bug was fixed in the package ubuntu-release-upgrader - 1:18.10.9

---------------
ubuntu-release-upgrader (1:18.10.9) cosmic; urgency=medium

  * data/release-upgrades: set Prompt to normal not lts
  * DistUpgradeQuirks.py: if ubuntu-desktop or snapd isn't in the package
    cache don't try and run the quirk for installing or upgrading snaps.
    (LP: #1787668)
  * DistUpgradeQuirks.py: Only refresh installed snaps to the ubuntu-18.10
    channel if they are tracking the ubuntu-18.04 channel. (LP: #1748581)

 -- Brian Murray <email address hidden> Tue, 28 Aug 2018 13:09:55 -0700

------- Comment From <email address hidden> 2018-08-29 03:35 EDT-------
I don't see a Bionic entry within the LP. I guess, this fix will also be applied for 18.04 LTS. Many thx in advance

I hope that it will be backported to 16.04 as well, without applying this fix to an upgrade to 18.04 isn't possible.

Thank you!

Hello errors.ubuntu.com, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I've tried normal do-release-upgrade xenial->bionic and it fails with keyerror on s390x.
Using the proposed upgrader, the upgrade is succeeding and starts the upgrade transaction correctly on s390x.
Therefore from s390x point of view this is fixed.

I went ahead and tested an upgrade from Ubuntu 16.04 to Ubuntu 18.04 using the release upgrader from -proposed without hacking the packages in the cache to ensure that the deb2snap migration (per Adam's suggestion) was still done. In /var/log/dist-upgrade/main.log we can see the following:

2018-08-29 10:53:25,381 DEBUG plugins for condition 'from_xenialPostUpgrade' are '[]'
2018-08-29 10:53:25,381 DEBUG quirks: running bionicPostUpgrade
2018-08-29 10:53:25,381 DEBUG running Quirks.bionicPostUpgrade
2018-08-29 10:54:03,616 DEBUG Install of snap gnome-3-26-1604 succeeded
2018-08-29 10:54:11,713 DEBUG Install of snap gtk-common-themes succeeded
2018-08-29 10:54:19,473 DEBUG Install of snap gnome-calculator succeeded
2018-08-29 10:54:27,876 DEBUG Install of snap gnome-characters succeeded
2018-08-29 10:54:36,738 DEBUG Install of snap gnome-logs succeeded
2018-08-29 10:54:45,931 DEBUG Install of snap gnome-system-monitor succeeded

So that verification is good too.

This bug was fixed in the package ubuntu-release-upgrader - 1:18.04.25

---------------
ubuntu-release-upgrader (1:18.04.25) bionic; urgency=medium

  * DistUpgradeQuirks.py: if ubuntu-desktop or snapd isn't in the package
    cache don't try and run the quirk for installing or upgrading snaps.
    (LP: #1787668)

 -- Brian Murray <email address hidden> Tue, 28 Aug 2018 15:46:57 -0700

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

------- Comment From <email address hidden> 2018-08-31 07:17 EDT-------
IBM Bugzilla status -> closed, Fix Released by Canonical