[SRU] update-notifier-common weekly cron job runs at the same time for all computers across the globe

Status changed to 'Confirmed' because the bug affects multiple users.

The design of cron.weekly is such that it would be wrong to vary its time of day, and it would be awkward to add a random sleep within the cron.weekly script itself for this (though we have done such things in the past). I think we will want to move this to a systemd timer unit instead.

We're still seeing regular request spikes to changelogs.ubuntu.com. Can we move release-upgrade-motd, the bit that downloads and checks for a new Ubuntu release to systemd timer units instead?

/usr/lib/ubuntu-release-upgrader/release-upgrade-motd which calls /usr/lib/ubuntu-release-upgrader/check-new-release which uses MetaReleaseCore from UpdateManager.Core.MetaRelease

This just happened with conntrack metrics below:

| * 20200920-0647
| 629385 dst=91.189.95.15
|
| * 20200920-0648
| 1444902 dst=91.189.95.15
|
| * 20200920-0649
| 1044261 dst=91.189.95.15

With request logs showing a surge in this:

| xxx.xxx.xxx.xxx - - [20/Sep/2020:06:46:34 +0000] "GET /meta-release-lts HTTP/1.1" 304 129 "-"
"Python-urllib/3.5"

Uploaded a test package to the following PPA which converts the two remaining cronjobs into systemd timers and tidies up a couple of other packaging bits:

https://launchpad.net/~waveform/+archive/ubuntu/update-notifier/+packages

Needs testing to ensure that the upgrade actually removes the existing cronjobs (couple of new maintscript bits *should* take care of this), and that the timings are reasonable.

In this version, only the motd timer has a calendar-based schedule (every Sunday, but smeared across 24h as requested). The package-data-downloader timer is started 5 minutes after system start, and then periodically each 24h after that, which roughly mimics what the old cronjob did under anacron (though there's also a dpkg trigger that may fire the task at arbitrary points too). I've added "After=" rules to both to try and ensure they're always scheduled after networking is available (using similar rules to the existing apt-daily timer), and service conditions which mimic the previous [ -x ] tests in the cronjobs.

This bug was fixed in the package update-notifier - 3.192.34

---------------
update-notifier (3.192.34) groovy; urgency=medium

  * Replace cronjobs with systemd timers. (LP: #1836475)
  * d/control:
    - Removed (now) redundant anacron dep
    - Bumped Standards-Version
    - Removed redundant autotools-dev dep and X-Python3-Version spec
  * d/update-notifier-common.maintscript:
    - Added rm_conffile for former cronjobs
    - Migrated rm_conffile cpu-checker bits from postinst/postrm/preinst
  * d/update-notifier-common.triggers:
    - Switched to interest-noawait as trigger is "non-crucial" (shouldn't
      block configuration of triggering packages)

 -- Dave Jones <email address hidden> Thu, 08 Oct 2020 13:32:55 +0100

Attaching patch for SRU to focal; test build is available from:

https://launchpad.net/~waveform/+archive/ubuntu/update-notifier/+packages

I've uploaded this to the Stable Release updates queue for Focal for review by a fellow SRU team member. Thanks for your contribution to Ubuntu!

Hello Junien, or anyone else affected,

Accepted update-notifier into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.30.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I tried to verify this but it looks like it was removed from focal-proposed.

> I tried to verify this but it looks like it was removed from focal-proposed.

Sorry, I should have communicated this better.

With the proposed patch for Focal, users who have customised the file in /etc/cron.* (eg. by removing it) won't have that customisation transferred to the systemd timer. This needs consideration, so we're removing it from the SRU so as not to block the SRU for bug 1901627.

Removed from focal-proposed

Any updates with regards to getting this backported to Focal?

> Sorry, I should have communicated this better.

> With the proposed patch for Focal, users who have customised the file in /etc/cron.* (eg. by
> removing it) won't have that customisation transferred to the systemd timer. This needs
> consideration, so we're removing it from the SRU so as not to block the SRU for bug 1901627.

Would it be sufficient to add some logic to the postinst disabling the new systemd timer if the existing cron-job didn't exist? Or add a warning message in the event that the existing cron-job didn't match some hash of a default?

I'm keen to try and avoid handling any more complex (arbitrary) customisations :)

Here are some regression scenarios:

Someone has taken control of all scheduled tasks on a load-critical server by either disabling cron, adjusting the cron.weekly job, or adjusting this particular weekly job in cron. Switching to a systemd timer and enabling it by default would disrupt such a user.

Someone doesn't want the outbound traffic (eg. on privacy grounds) and has disabled the job via cron. Again, this might have happened at various points in cron's configuration.

> I'm keen to try and avoid handling any more complex (arbitrary) customisations

I agree! I think I prefer adding a random sleep for the SRU for this reason, even if the appropriate fix for the development release that landed in Groovy was to switch to a systemd timer. However a long random sleep in cron.weekly will hold up other cron.weekly jobs, so given that /usr/lib/ubuntu-release-upgrader/release-upgrade-motd already gates with a timestamp, perhaps backgrounding a shorter random delay would be preferable if that would be acceptable to Canonical IS?

I would want to refer to other SRU team members before making a decision. If somebody is opposed to the random delay method, a systemd timer can still work, but in that case breaking the above use cases should be deliberate and documented.

On Mon, Jun 14, 2021 at 04:16:28PM -0000, Robie Basak wrote:
> > I'm keen to try and avoid handling any more complex (arbitrary)
> customisations
>
> I agree! I think I prefer adding a random sleep for the SRU for this
> reason, even if the appropriate fix for the development release that
> landed in Groovy was to switch to a systemd timer. However a long random
> sleep in cron.weekly will hold up other cron.weekly jobs, so given that
> /usr/lib/ubuntu-release-upgrader/release-upgrade-motd already gates with
> a timestamp, perhaps backgrounding a shorter random delay would be
> preferable if that would be acceptable to Canonical IS?
>

We could try with a shorter random delay and see if it helps spread
the load.

The 'update-notifier-common' cron.weekly job is usually last so
wouldn't really hold up that many other cron.weekly jobs. If a shoter
random delay/sleep doesn't help, we could consider renaming it to
something like 'zzupdate-notifier-common' and increasing the delay
then.

Attaching a debdiff for the focal version which implements a random delay of up to 1 hour. The delay is executed in the background to ensure that other cronjobs are not held up. Hopefully 1 hour is sufficient: it's considerably shorter than the 24 hours that the systemd timer "smears" this action over, but mitigates the issue of an update being missed because the server is shutdown / rebooted during the delay.

Test build available in the following PPA:

https://launchpad.net/~waveform/+archive/ubuntu/update-notifier

@waveform, perfect, thanks!

I've sponsored this for Dave.

Hello Junien, or anyone else affected,

Accepted update-notifier into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.30.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Looks good to me:

| ubuntu@juju-87625f-hloeung-100:/etc/apt$ sudo apt-get install update-notifier-common
| Reading package lists... Done
| Building dependency tree
| Reading state information... Done
| The following packages were automatically installed and are no longer required:
| dnsmasq-base kpartx libfl2 libidn11 libsgutils2-2 liburcu6 sg3-utils sg3-utils-udev
| Use 'sudo apt autoremove' to remove them.
| The following packages will be upgraded:
| update-notifier-common
| 1 upgraded, 0 newly installed, 0 to remove and 44 not upgraded.
| Need to get 133 kB of archives.
| After this operation, 0 B of additional disk space will be used.
| Get:1 http://us.archive.ubuntu.com/ubuntu focal-proposed/main amd64 update-notifier-common all 3.192.30.9 [133 kB]
| Fetched 133 kB in 0s (3458 kB/s)
| (Reading database ... 130802 files and directories currently installed.)
| Preparing to unpack .../update-notifier-common_3.192.30.9_all.deb ...
| Unpacking update-notifier-common (3.192.30.9) over (3.192.30.8) ...
| Setting up update-notifier-common (3.192.30.9) ...
| Installing new version of config file /etc/cron.weekly/update-notifier-common ...

Comparing it with the old backup copy:

| ubuntu@juju-87625f-hloeung-100:/etc/apt$ diff -Naurp ~/update-notifier-common /etc/cron.weekly/update-notifier-common
| --- /home/ubuntu/update-notifier-common 2021-08-13 05:55:55.859177053 +0000
| +++ /etc/cron.weekly/update-notifier-common 2021-08-05 14:01:11.000000000 +0000
| @@ -4,5 +4,12 @@ set -e
|
| [ -x /usr/lib/ubuntu-release-upgrader/release-upgrade-motd ] || exit 0
|
| -# Check to see whether there is a new version of Ubuntu available
| -/usr/lib/ubuntu-release-upgrader/release-upgrade-motd
| +sleep_then_check() {
| + # Sleep for up to an hour to spread the load of checking for updates on
| + # the Ubuntu infrastructure
| + sleep $(shuf -i 1-3600 -n 1)
| + # Check to see whether there is a new version of Ubuntu available
| + /usr/lib/ubuntu-release-upgrader/release-upgrade-motd
| +}
| +
| +sleep_then_check &

| ubuntu@juju-87625f-hloeung-100:/etc/cron.weekly$ apt-cache policy update-notifier-commonupdate-notifier-common:
| Installed: 3.192.30.9
| Candidate: 3.192.30.9
| Version table:
| *** 3.192.30.9 500
| 500 http://us.archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
| 100 /var/lib/dpkg/status
| 3.192.30.8 500
| 500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
| 3.192.30 500
| 500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages

Thanks Brian (brian-murray) for sponsoring this change.

The verification of the Stable Release Update for update-notifier has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package update-notifier - 3.192.30.9

---------------
update-notifier (3.192.30.9) focal; urgency=medium

  * Sleep for up to an hour in the background in order to spread the load
    of checking for release updates on the Ubuntu infrastructure
    (LP: #1836475)

 -- Dave Jones <email address hidden> Thu, 05 Aug 2021 15:01:11 +0100

Hello Junien, or anyone else affected,

Accepted update-notifier into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.1.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

LGTM, thanks!

| ubuntu@juju-87625f-hloeung-66:~$ cp /etc/cron.weekly/update-notifier-common ~/
| ubuntu@juju-87625f-hloeung-66:~$ sudo apt-get update
| ...
| Get:8 http://us.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages [104 kB]
| ...
| ubuntu@juju-87625f-hloeung-66:~$ sudo apt-get install update-notifier-common
| ubuntu@juju-87625f-hloeung-66:~$ diff -Naurp ~/update-notifier-common /etc/cron.weekly/update-notifier-common
| --- /home/ubuntu/update-notifier-common 2021-08-26 00:27:43.687383953 +0000
| +++ /etc/cron.weekly/update-notifier-common 2021-08-23 22:33:23.000000000 +0000
| @@ -4,5 +4,12 @@ set -e
|
| [ -x /usr/lib/ubuntu-release-upgrader/release-upgrade-motd ] || exit 0
|
| -# Check to see whether there is a new version of Ubuntu available
| -/usr/lib/ubuntu-release-upgrader/release-upgrade-motd
| +sleep_then_check() {
| + # Sleep for up to an hour to spread the load of checking for updates on
| + # the Ubuntu infrastructure
| + sleep $(shuf -i 1-3600 -n 1)
| + # Check to see whether there is a new version of Ubuntu available
| + /usr/lib/ubuntu-release-upgrader/release-upgrade-motd
| +}
| +
| +sleep_then_check &
| ubuntu@juju-87625f-hloeung-66:~$ sudo /etc/cron.weekly/update-notifier-common
| ubuntu@juju-87625f-hloeung-66:~$ ps afxuww | grep sleep
| root 22110 0.0 0.0 7932 740 pts/0 S 00:29 0:00 \_ sleep 767
| ubuntu@juju-87625f-hloeung-66:~$ ls -la /var/lib/ubuntu-release-upgrader/release-upgrade-available
| -rw-r--r-- 1 root root 80 Aug 26 00:27 /var/lib/ubuntu-release-upgrader/release-upgrade-available
| ubuntu@juju-87625f-hloeung-66:~$ New release '20.04.2 LTS' available.
| Run 'do-release-upgrade' to upgrade to it.
|

The additional output from the end is when it ran, LP:1940011 is to see if we can silence that to reduce weekly cron spam.

This bug was fixed in the package update-notifier - 3.192.1.12

---------------
update-notifier (3.192.1.12) bionic; urgency=medium

  * Sleep for up to an hour in the background in order to spread the load
    of checking for release updates on the Ubuntu infrastructure
    (LP: #1836475)

 -- Brian Murray <email address hidden> Mon, 23 Aug 2021 15:33:27 -0700