Update notifier keeps popping up even when there is nothing to do

I can confirm this bug. I also use Ubuntu 20.04. I noticed, that it happens every time, when apt has an update, that isn't visible through the graphical frontend updater. If this is the case, the update notifier pops up with the message "System is up to date". If I than do "sudo apt update", apt shows me, that there is an update available. If I install this "invisible" update, the update notifier is calm again, e.g. the message "System is up to date" disappears.
This problem started after the introduction of Ubuntu Pro. It must have something to do with it.
Please, fix this bug.

Status changed to 'Confirmed' because the bug affects multiple users.

This is not directly related to Ubuntu Pro.

Those 'invisible' updates are actually phased updates.
Update notifier is right to poke because there are updates, but it does not show it because those updates are phased for the GUI.
There is logic to prevent that in the code and it was working before, so I can confirm this is a regression.

I am working on it.

Thank you very much for your answer and your effort, to solve this problem.

This bug was fixed in the package update-notifier - 3.192.63

---------------
update-notifier (3.192.63) lunar; urgency=medium

  * Isolate creation of the esm apt cache in apt-check (LP: #2008212)

 -- Renan Rodrigo Barbosa <email address hidden> Mon, 27 Feb 2023 23:15:02 -0300

So this change/fix actually seems to inadvertently fix #2007311. All along I thought it was a problem with upstream cache being broken, but I could definitely see it being plausible that phased updates could cause this same weird behavior.

Anyone who writes anything that depends on apt-check may be in for nasty surprises when apt-check breaks and returns bad data instead of predictably formatted pairs of integers.

Example:
 /usr/lib/update-notifier/apt-check
E: Unknown Error: '<class 'KeyError'>' ("The cache has no package named 'libxmlb2:amd64'")

/usr/lib/update-notifier/apt-check --human-readable
E: Unknown Error: '<class 'KeyError'>' ("The cache has no package named 'libxmlb2:amd64'")

Interesting! It's rare to see a side-effect fix instead of a side-effect bug :|
Thanks for checking it.

There are plans to review all those cache-based scripts across packages in the near future, so this kind of error gets more and more under control.

Fantastic - hoping that your minor change to fix apt-check breezes through review and gets released for Focal and Jammy ASAP.

Status changed to 'Confirmed' because the bug affects multiple users.

Hello FM33, or anyone else affected,

Accepted update-notifier into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.54.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello FM33, or anyone else affected,

Accepted update-notifier into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.30.17 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello FM33, or anyone else affected,

Accepted update-notifier into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.1.19 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello FM33, or anyone else affected,

Accepted update-notifier into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.168.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Tested and confirmed on Focal server to fix my issue.

I don't currently have a misbehaving Jammy server to test on but I have no reason to believe that this fix wouldn't work the same way on Jammy, since I was using a local copy from the ubuntu/devel tree earlier this week on a broken Jammy system (which has since been fixed, so I can't re-test...)

Verification for focal:

apt show python3-software-properties 2>&1 | grep Phase
Phased-Update-Percentage: 0

# Currently installed version:
apt-cache policy update-notifier-common | head -n 3
update-notifier-common:
  Installed: 3.192.30.16
  Candidate: 3.192.30.17

/usr/lib/update-notifier/apt-check -p 2>&1 | grep python
python3-software-properties

# after upgrade to -proposed:
apt-cache policy update-notifier-common | head -n 3
update-notifier-common:
  Installed: 3.192.30.17
  Candidate: 3.192.30.17

/usr/lib/update-notifier/apt-check -p 2>&1 | grep python
#

# phased out python3-software-properties not listed

Behavior consistent with the test plan.

Verification for bionic:

apt-cache policy python3.6
python3.6:
  Installed: 3.6.9-1~18.04ubuntu1.9
  Candidate: 3.6.9-1~18.04ubuntu1.10
  (...)

I've decided to fake "phased updates" for python3.6.

apt show python3.6 2>&1 | grep Phased
Phased-Update-Percentage: 0

/usr/lib/update-notifier/apt-check -p 2>&1 | grep -w python3.6
python3.6
python3.6-minimal

After upgrading to -proposed the Phased-Update-Percentage packages are still listed.

I will be looking into this. Maybe I am doing something incorrectly with "faking" the phased status.

I just added the "Phased-Update-Percentage: 0" line to python3.6 entries in archive.ubuntu.com_ubuntu_dists_bionic-updates_main_binary-amd64_Packages and security.ubuntu.com_ubuntu_dists_bionic-security_main_binary-amd64_Packages

(reusing the template from above)

Verification for xenial:

apt show ubuntu-advantage-tools 2>&1 | grep Phase
Phased-Update-Percentage: 10

# Currently installed version:
apt-cache policy update-notifier-common | head -n 3
update-notifier-common:
  Installed: 3.168.19
  Candidate: 3.168.20

/usr/lib/update-notifier/apt-check -p 2>&1 | grep advantage
ubuntu-advantage-tools

# after upgrade to -proposed:
apt-cache policy update-notifier-common | head -n 3
update-notifier-common:
  Installed: 3.168.20
  Candidate: 3.168.20

/usr/lib/update-notifier/apt-check -p 2>&1 | grep advantage
#

# phased out ubuntu-advantage-tools not listed

Behavior consistent with the test plan.

I am observing behavior similar to bionic on jammy. I am looking into the whole "faking" process to compare if it's consistent with what I observed on focal with actual phased updates.

I tried the same here - it seems just a matter of choice of package;
python3.6 is a security update, and those are never phased, so even forcing it won't work.

I simulated it with `parted` (from proposed) and I see it works on Bionic.

Verification for bionic:

apt show parted 2>&1 | grep Phase
Phased-Update-Percentage: 0

# Currently installed version:
apt-cache policy update-notifier-common | head -n 3
update-notifier-common:
  Installed: 3.192.1.18
  Candidate: 3.192.1.19

/usr/lib/update-notifier/apt-check -p 2>&1 | grep parted
libparted2
parted

# after upgrade to -proposed:
apt-cache policy update-notifier-common | head -n 3
update-notifier-common:
  Installed: 3.192.1.19
  Candidate: 3.192.1.19

/usr/lib/update-notifier/apt-check -p 2>&1 | grep advantage
libparted2
#

# phased out parted not listed

Behavior consistent with the test plan.

Verification for jammy:
As mentioned in the bug description, this bug happens on focal and earlier.
On Jammt, apt already has the phasing implemented, and update-notifier will not fail even with the bug being present.

apt show apt 2>&1 | grep Phase
# nothing faked yet

/usr/lib/update-notifier/apt-check -p 2>&1 | grep apt
apt
libapt-pkg6.0
apt-utils

(Faking phasing on -proposed apt package)

apt show apt 2>&1 | grep Phase
Phased-Update-Percentage: 0

# Currently installed version:
apt-cache policy update-notifier-common | head -n 3
update-notifier-common:
  Installed: 3.192.54.5
  Candidate: 3.192.54.6

/usr/lib/update-notifier/apt-check -p 2>&1 | grep apt
libapt-pkg6.0

# after upgrade to -proposed:
apt-cache policy update-notifier-common | head -n 3
update-notifier-common:
  Installed: 3.192.54.6
  Candidate: 3.192.54.6

/usr/lib/update-notifier/apt-check -p 2>&1 | grep apt
libapt-pkg6.0
#

# same behavior, was respecting the phasing even before.

Thanks for explaining Renan, it makes sense and I can confirm results as expected after switching to a package outside of -security.

This bug was fixed in the package update-notifier - 3.192.54.6

---------------
update-notifier (3.192.54.6) jammy; urgency=medium

  * Isolate creation of the esm apt cache in apt-check (LP: #2008212)

 -- Renan Rodrigo Barbosa <email address hidden> Mon, 27 Feb 2023 23:15:02 -0300

The verification of the Stable Release Update for update-notifier has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package update-notifier - 3.192.30.17

---------------
update-notifier (3.192.30.17) focal; urgency=medium

  * Isolate creation of the esm apt cache in apt-check (LP: #2008212)

 -- Renan Rodrigo Barbosa <email address hidden> Mon, 27 Feb 2023 23:15:02 -0300

This bug was fixed in the package update-notifier - 3.192.1.19

---------------
update-notifier (3.192.1.19) bionic; urgency=medium

  * Isolate creation of the esm apt cache in apt-check (LP: #2008212)

 -- Renan Rodrigo Barbosa <email address hidden> Mon, 27 Feb 2023 23:15:02 -0300

This bug was fixed in the package update-notifier - 3.168.20

---------------
update-notifier (3.168.20) xenial; urgency=medium

  * Isolate creation of the esm apt cache in apt-check (LP: #2008212)

 -- Renan Rodrigo Barbosa <email address hidden> Mon, 27 Feb 2023 23:15:02 -0300