zypper often fails to download packages when using HTTPS

debdiff attached

For Mantic the fix can come from the new upstream version that I'll work to get in Debian unstable shortly

The attachment "debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Thanks for the patch Luca! Could you tell us when are you planning to fix this in Debian unstable and therefore Mantic? We should get this fixed in the development release before it reaches the SRU team.

Moreover, there might be a versioning issue with your proposed debdiff. You are using version 17.25.7-2.1ubuntu0.22.04.1 and in Kinetic (still supported until end of July) we have 17.25.7-2.1, ideally, we should be fixing the bug in both of them. But in the case we decide to skip Kinetic, your version is greater than what we have there, and it wouldn't allow users to upgrade properly. I do understand that the Kinetic EOL is coming but for now it is still supported. So I think we should fix both releases (Jammy and Kinetic) , or at least use another version string, such as 17.25.7-2.1~ubuntu0.22.04.1.

And also what about Lunar? I suppose it is also affected, right? We can't fix Jammy and leave the newer stable releases (> Jammy) affected by the bug.

> Thanks for the patch Luca! Could you tell us when are you planning to fix this in Debian unstable and therefore Mantic? We should get this fixed in the development release before it reaches the SRU team.

It's done now, newest libsolv/libzypp/zypper are in unstable.

> So I think we should fix both releases (Jammy and Kinetic) , or at least use another version string, such as 17.25.7-2.1~ubuntu0.22.04.1.

I'll fix the version string, it's ok for this to wait until it's EOL to save some work and time, given it's just a couple of weeks.

> And also what about Lunar? I suppose it is also affected, right? We can't fix Jammy and leave the newer stable releases (> Jammy) affected by the bug.

Ok, I'll get a debdiff and a quick test going for Lunar as well as soon as I can.

Thanks Luca!

I will unsubscribe ubuntu-sponsors from this bug for now. Please, subscribe it back as soon as you believe this is ready for another review or an upload!

For Lunar:

root@lunar:/tmp# dpkg -l | grep libzypp
ii libzypp-bin 17.25.7-2.3ubuntu0.23.04.1 amd64 openSUSE/SLES package management system library (library tools)
ii libzypp-common 17.25.7-2.3ubuntu0.23.04.1 all openSUSE/SLES package management system library (common files)
ii libzypp-config 17.25.7-2.3ubuntu0.23.04.1 all openSUSE/SLES package management system library (configuration)
ii libzypp1722:amd64 17.25.7-2.3ubuntu0.23.04.1 amd64 openSUSE/SLES package management system (library)
ii zypper 1.14.42-2 amd64 command line software manager using libzypp
ii zypper-common 1.14.42-2 all command line software manager using libzypp (common files)
root@lunar:/tmp# cat /etc/os-release
PRETTY_NAME="Ubuntu 23.04"
NAME="Ubuntu"
VERSION_ID="23.04"
VERSION="23.04 (Lunar Lobster)"
VERSION_CODENAME=lunar
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=lunar
LOGO=ubuntu-logo
root@lunar:/tmp# mkosi -d opensuse -r tumbleweed --format=directory --package=systemd build
‣ Removing output files…
‣ Detaching namespace
‣ Setting up temporary workspace.
‣ Temporary workspace set up in /tmp/.mkosi-fl4ofptb
‣ Running second (final) stage…
‣ Mounting image…
‣ Setting up basic OS tree…
‣ Mounting Package Cache
‣ Installing openSUSE…
Adding repository 'repo-oss' ................................................................................................................................................................................[done]
Repository 'repo-oss' successfully added

URI : http://download.opensuse.org/tumbleweed/repo/oss/
Enabled : Yes
GPG Check : Yes
Autorefresh : No
Priority : 99 (default priority)

Repository priorities are without effect. All enabled repositories share the same priority.
Adding repository 'repo-update' .............................................................................................................................................................................[done]
Repository 'repo-update' successfully added

URI : http://download.opensuse.org/update/tumbleweed/
Enabled : Yes
GPG Check : Yes
Autorefresh : No
Priority : 99 (default priority)

Repository priorities are without effect. All enabled repositories share the same priority.
‣ Mounting API VFS…

Automatically importing the following key:

  Repository: repo-oss
  Key Name: openSUSE Project Signing Key <email address hidden>
  Key Fingerprint: AD485664 E901B867 051AB15F 35A2F86E 29B700A4
  Key Created: Mon Jun 20 15:03:14 2022
  Key Expires: Fri Jun 19 15:03:14 2026
  Rpm Name: gpg-pubkey-29b700a4-62b07e22

Building repository 'repo-oss' cache .................................................................

For Lunar another backport was needed from Mantic, to fix a build failure due a to a missing header.

> or at least use another version string, such as 17.25.7-2.1~ubuntu0.22.04.1.

That would mean the update is actually not installed on jammy, given 17.25.7-2.1 >> 17.25.7-2.1~ubuntu0.22.04.1
I am fine with waiting until Kinetic is EOL. Alternatively, simply change the distro from jammy to kinetic in the debdiff, the current versions are identical.

kinetic is now EOL, mantic was fixed (via debian sync), and the tested debdiff is provided for lunar and jammy, so adding back the sponsors alias

Hey, seeing this in the queue again.

Lunar:
- version is ok now
- changelog is ok
  - we wouldn't need the "Non-maintainer upload" but that is still ok
  - I'd prefer to mention the related patch file in the CL
  - overall this keeps/matches the Debian CL style and that is fine to keep as-is IMHO
- patches have headers and match what is in the latest version in upstream, mantic and debian

Jammy:
- here the long version with the release wouldn't be needed (only if adjacent releases have the same version), but it isn't hurting either
- changelog is ok (just like in lunar)
- patches have headers and match what is in the latest version in upstream, mantic and debian

The SRU template was already good, but I made a minor change to the regression risk.
If clear from the changes it is always preferred to point to which use-case, action or step an error would be. in This case all changes are related to file transfer.

I'm adding the change to maintainer.
It is a bit odd as a friendly Debian maintainer makes me add "Ubuntu Developers", but it is true that it is no more the original maintainer. So we have to follow the policy and the CL correctly states Luca as the good guy providing this :-)

FYI: any upload that is the first deviating from Debian needs to run `update-maintainer` to reflect that it is no more identical with the sync from Debian.

Double checked the resulting diff in .dsc before upload, LGTM still.
Uploaded both to -unapproved for the SRU team to have a look.

Thank you!

I'm unsubscribing ~ubuntu-sponsors as these were upload to -unapproved.

+ * Non-maintainer upload.

Well, that's incorrect in an Ubuntu SRU. Reuploading to adjust the changelog.

> Verified that zypper still works on jammy to bootstrap a suse image using mkosi:

This does not appear to explain how to reproduce the problem that this is here to fix, and to verify that the new package fixes it.

It's a race condition, repro rate is not high, but it hits hard if you have a CI that uses it many times an hour. Like Github Actions which uses Jammy images.
Try enough times to install packages from https repos on Jammy with zypper, and eventually you'll hit it. Not very useful to verify individual fixes of course, but these are all backports from upstream.

What I can offer is that once it hits proposed, I can enabled it on the CI system on Github Actions where we saw the problem, and turn HTTPS back on. Given a week or so, that should confirm whether this helps or not, and also if it introduces new issues. Would that work?

I tried this command in jammy, but that is using http, not https, for the downloads, as far as I can see:

# mkosi -d opensuse -r tumbleweed --format=directory --package=systemd build
root@j:~# time mkosi -d opensuse -r tumbleweed --format=directory --package=systemd build
‣ Detaching namespace
‣ Setting up package cache…
‣ Setting up package cache /root/.mkosi-psvpik4e complete
‣ Setting up temporary workspace.
‣ Temporary workspace set up in /root/.mkosi-hrg8e6wg
‣ Running second (final) stage…
‣ Mounting image…
‣ Setting up basic OS tree…
‣ Mounting Package Cache
‣ Installing openSUSE…
Adding repository 'repo-oss' ............................................................................................................................[done]
Repository 'repo-oss' successfully added

URI : http://download.opensuse.org/tumbleweed/repo/oss/
(...)

Confirmed via ss:
tcp ESTAB 0 214 10.0.102.5:43674 195.135.221.134:http
tcp ESTAB 0 0 10.0.102.5:52700 200.236.31.10:http
tcp ESTAB 0 0 10.0.102.5:33796 200.9.155.223:http

In order for this to be a useful test case, it needs to perform these downloads over https, no?

We had to disable https by default because of this issue. The option --mirror https://download.opensuse.org can be used to restore it:

root@jammy:~# mkosi -d opensuse -r tumbleweed --format=directory --package=systemd --mirror=https://download.opensuse.org build
‣ Detaching namespace
‣ Setting up package cache…
‣ Setting up package cache /root/.mkosi-zj594cmr complete
‣ Setting up temporary workspace.
‣ Temporary workspace set up in /root/.mkosi-laml1y9f
‣ Running second (final) stage…
‣ Mounting image…
‣ Setting up basic OS tree…
‣ Mounting Package Cache
‣ Installing openSUSE…
Adding repository 'repo-oss' ...................................................................................[done]
Repository 'repo-oss' successfully added

URI : https://download.opensuse.org/tumbleweed/repo/oss/
Enabled : Yes
GPG Check : Yes
Autorefresh : No
Priority : 99 (default priority)

Repository priorities are without effect. All enabled repositories share the same priority.
Adding repository 'repo-update' ................................................................................[done]
Repository 'repo-update' successfully added

URI : https://download.opensuse.org/update/tumbleweed/
Enabled : Yes
GPG Check : Yes
Autorefresh : No
Priority : 99 (default priority)

Repository priorities are without effect. All enabled repositories share the same priority.
‣ Mounting API VFS

Automatically importing the following key:

  Repository: repo-oss
  Key Name: openSUSE Project Signing Key <email address hidden>
  Key Fingerprint: AD485664 E901B867 051AB15F 35A2F86E 29B700A4
  Key Created: Mon Jun 20 15:03:14 2022
  Key Expires: Fri Jun 19 15:03:14 2026
  Rpm Name: gpg-pubkey-29b700a4-62b07e22

Building repository 'repo-oss' cache ...........................................................................[done]
Building repository 'repo-update' cache ........................................................................[done]
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following 89 NEW packages are going to be installed:
  aaa_base alts bash bash-sh branding-openSUSE busybox busybox-grep busybox-sed busybox-xz chkstat
  compat-usrmerge-tools coreutils crypto-policies dbus-1 dbus-1-common dbus-1-daemon dbus-1-tools diffutils file-magic
  filesystem fillup findutils glibc kbd kbd-legacy libacl1 libalternatives1 libapparmor1 libattr1 libaudit1 libblkid1
  libbz2-1 libcap-ng0 libcap2 libcrypt1 libdbus-1-3 libeconf0 libexpat1 libfdisk1 libgcc_s1 libgcrypt20 libgmp10
  libgpg-error0 libip4tc2 libkmod2 liblz4-1 liblzma5 libmagic1 libmount1 libncurses6 libnss_usrfiles2 libopenssl3
  libpcre2-8-0 libpkgconf3 libreadline8 libseccomp2 libselinux1 libsepol2 libsmartcols1 libstdc++6 libsystemd0
  libutempter0 libuuid1 libz-ng-compat1 libzstd1 ncurses-utils netcfg openSUSE-build-key openSUSE-release
  openSUSE-release-appliance-custom pam pam-config patterns-base-minimal_base patterns-glibc-hwcaps-x86_64_v3
  permissions permissions-config pkgconf pkgconf-m4 pkgconf-pkg-config system-group-hardware system-user-root systemd
  systemd-default-settin...

I really wish we had a better test case. I left mkosi running for hours in a loop, using https, and there was no single download failure yet.

Would you be able to provide a CI result showing this https curl failure with the packages from jammy or lunar, and then let it loose again with the proposed packages when they are accepted, to show that in the same time frame there are no new failures?

Or can you perhaps think of another test case, perhaps simpler and quicker, like using this library in a localhost context with something served by apache over https, doesn't have to be a real repo, but just something that would exercise the https code path and that we could leave running for a while?

Unfortunately logs are pruned from Github Actions after a month or so, and we disabled https longer than that. And I really don't know how to have a unit test.
However as mentioned, once the update is in proposed I can certainly re-enable https and update to proposed, and report back.

@bluca thanks for the offer. I think that's the best we can get. If you can run the proposed package in your ci with https enabled, where previously you were seeing errors, that would help. If there are no errors after "some time", that would show us:
a) no regressions
b) https is working again

With that test plan, +1 on accepting this. But please be sure to use the package from proposed, and to enable https :)

Thanks, I'll do it in two stages: first, enable proposed but leave https off for a few days, to ensure there are no regressions in the base case. Then, after a few days if there's no regression, enable https, and monitor to see if it makes the situation better/by how much/etc.

Hello Luca, or anyone else affected,

Accepted libzypp into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libzypp/17.25.7-2.3ubuntu0.23.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Luca, or anyone else affected,

Accepted libzypp into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libzypp/17.25.7-2.1ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Ah it's there too, just saw the first message, never mind - thanks!

jammy-proposed is enabled and working fine in the CI: https://github.com/systemd/mkosi/pull/1856
around mid-week next week I'll also re-enable https

jammy-proposed has been working fine with http. I have now enabled https: https://github.com/systemd/mkosi/commit/28d2bb0fe10ce7ba0d948c7f42c0b116f2180611
I will report back at the end of the week again.

This has been running on jammy in the mkosi CI for a few days now, with https, and while before failures happened on almost every run, there hasn't been a single instance since the update, so I am satisfied that there are no regressions and also that the problems are fixed, changing labels accordingly.

I verified the jammy job logs, and confirmed that they used the libzypp version from jammy-proposed. Note that you enabled proposed as a whole, so you also got other proposed updates in your test runs, not just libzypp. That's fine for this SRU, but you risked introducing other bugs in your CI ;)

You also flipped the verification tag for lunar, did you exercise libzypp from lunar-proposed as well in your ci, or just jammy?

The CI can only run on Jammy, I have checked that zypper with https works in Lunar manually in a local VM

Thanks, I understand you can't just switch your CI to run on lunar, and you helped a lot with this bug already. I also tested this on lunar again and see no issues.

This bug was fixed in the package libzypp - 17.25.7-2.3ubuntu0.23.04.1

---------------
libzypp (17.25.7-2.3ubuntu0.23.04.1) lunar; urgency=medium

  [ Adrian Bunk ]
  * Fix FTBFS due to missing #include. (Closes: #1028680)

  [ Luca Boccassi ]
  * Backport patches to improve zypper reliability with https
    (LP: #2025400)

 -- Luca Boccassi <email address hidden> Sun, 09 Jul 2023 20:28:00 +0100

The verification of the Stable Release Update for libzypp has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package libzypp - 17.25.7-2.1ubuntu0.22.04.1

---------------
libzypp (17.25.7-2.1ubuntu0.22.04.1) jammy; urgency=medium

  * Backport patches to improve zypper reliability with https
    (LP: #2025400)

 -- Luca Boccassi <email address hidden> Sat, 01 Jul 2023 15:35:26 +0100