zlib: compressBound() returns an incorrect result on z15

Bug #1961427 reported by bugproxy
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
Undecided
Skipper Bug Screeners
bedtools (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Invalid
Undecided
Unassigned
Impish
Won't Fix
Undecided
Unassigned
Jammy
Fix Released
Undecided
Brian Murray
htslib (Ubuntu)
Invalid
Undecided
Unassigned
Focal
Fix Released
Undecided
Brian Murray
Impish
Won't Fix
Undecided
Unassigned
Jammy
Invalid
Undecided
Unassigned
zlib (Ubuntu)
Fix Released
High
Brian Murray
Focal
Fix Released
Undecided
Brian Murray
Impish
Won't Fix
High
Unassigned
Jammy
Fix Released
High
Brian Murray

Bug Description

SRU Justification:
==================

[Impact]

* zlib: compressBound() returns an incorrect result on IBM z15 hardware.

* Passing the result of compressBound() to compress() results
  in an error code.

* This is because compressBound() is not adjusted for DFLTCC.

[Fix]

* Adjust compressBound() for DFLTCC like it's already done
  for deflateBound().

* Since zlib project does not accept patches at the moment,
  the fix has been integrated into the DFLTCC pull request:
  https://github.com/madler/zlib/pull/410
  The commitid is b25781e735363e04f6c56e21431c47e4afc50b17.

* The fix extracted out of the above is:
  https://launchpadlibrarian.net/589857296/debdiff_zlib_1.2.11.dfsg-2ubuntu7_to_zlib_1.2.11.dfsg-2ubuntu8_jammy.diff

* On top of this actual zlib fix, there is another patch needed:
 'Remove compressBound assertions. (PR #1258)' for htslib.

* But there is a standalone 'htslib' package version, as well as a
  htslib version included in (some) 'bedtools' packages.
  Both need to be patched (see '[Other]' for more details).

[Test Plan]

* An IBM z15 system (LPAR, z/VM guest or KVM virtual machine)
  with Ubuntu Server 21.10 (or 22.04).

* A test can be done based on the following C test program:
      #include <assert.h>
      #include <stdlib.h>
      #include <zlib.h>
      int main() {
          Bytef in_buf[128], out_buf[1024];
          for (size_t i = 0; i < sizeof(in_buf); i++)
              in_buf[i] = rand();
          uLongf dest_len = compressBound(sizeof(in_buf));
          assert(dest_len <= sizeof(out_buf));
          int ret = compress(out_buf, &dest_len,
                             in_buf, sizeof(in_buf));
          assert(ret == Z_OK);
          }

* The test needs to be done by IBM, due to the requirements
  for the special z15 hardware.

* A successful test was just completed, based on the version in jammy-proposed, which is at the same code level that the impish version this SRU is targeted for.

[Where problems could occur]

* If the adjustment of compressBound() for DFLTCC is done
  erroneously the issue can still be present or in worst case
  even affect Z systems other than z15 only.

* The compression can become errorneous with the new changes,
  e.g. in compressBound.

* Mistakes in dfltcc_free_window OF and especially DEFLATE_BOUND_COMPLEN,
  (incl. the bit definitions), may cause various and unforseen defects.

* Any build time issues that might have been introduced by this patch
  can be identified by a test build; this was done and is available here:
  https://launchpad.net/~fheimes/+archive/ubuntu/lp1961427

[Other Info]

* Ubuntu jammy, impish and focal are affected by the zlib issue.

* The 'htslib' version '1.13+ds' (as it is part of I, J and K),
  already includes the patch,
  hence only htslib '1.10.2' in focal needs to be patched.

* The bedtools version '2.30.0+dfsg' (as it is part of I, J and K),
  requires the patch,
  but version '2.27.1+dfsg' bedtools in focal does not incl. an
  embedded htslib, hence does not need to be (actually can't be)
  patched.

* Patched version of the affected htslib and bedtools packages
  are build and also available at this PPA:
  https://launchpad.net/~fheimes/+archive/ubuntu/lp1961427

__________

Description: zlib: compressBound() returns an incorrect result on z15
Symptom: Passing the result of compressBound() to compress()
               results in an error code.
Problem: compressBound() is not adjusted for DFLTCC.
Solution: Adjust compressBound() for DFLTCC like it's already done
               for deflateBound(). Since zlib project does not accept
               patches at the moment, the fix has been integrated into
               the DFLTCC pull request:
               https://github.com/madler/zlib/pull/410
               The commitid is b25781e735363e04f6c56e21431c47e4afc50b17.

Reproduction: z15 only:
               #include <assert.h>
               #include <stdlib.h>
               #include <zlib.h>
               int main() {
                   Bytef in_buf[128], out_buf[1024];
                   for (size_t i = 0; i < sizeof(in_buf); i++)
                       in_buf[i] = rand();
                   uLongf dest_len = compressBound(sizeof(in_buf));
                   assert(dest_len <= sizeof(out_buf));
                   int ret = compress(out_buf, &dest_len,
                                      in_buf, sizeof(in_buf));
                   assert(ret == Z_OK);
               }

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-194923 severity-high targetmilestone-inin2110
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Frank Heimes (fheimes)
affects: linux (Ubuntu) → zlib (Ubuntu)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
tags: added: fr-2079
Revision history for this message
Frank Heimes (fheimes) wrote :

debdiff for zlib_1.2.11.dfsg-2ubuntu7 to zlib_1.2.11.dfsg-2ubuntu8
(except the of the top line of the changelog:
zlib (1:1.2.11.dfsg-2ubuntu8) impish; urgency=medium
the debdiff is the same for impish and jammy)

  * d/p/410-lp1961427.patch ported from zlib #410, fixing
    compressBound() with hw acceleration. LP: #1961427
    Refreshed some quilt patches just to remove offsets.
    Thanks to Ilya Leoshkevich <email address hidden>.

summary: - [Ubuntu 21.10] zlib: compressBound() returns an incorrect result on z15
+ zlib: compressBound() returns an incorrect result on z15
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "zlib_1.2.11.dfsg-2ubuntu7_to_zlib_1.2.11.dfsg-2ubuntu8 (jammy)" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Mathew Hodson (mhodson)
Changed in zlib (Ubuntu Impish):
importance: Undecided → High
Changed in zlib (Ubuntu Jammy):
importance: Undecided → High
Revision history for this message
Brian Murray (brian-murray) wrote (last edit ):

If this needs SRU'ing to impish please update the bug description accordingly. I've sponsored this for Jammy.

Changed in zlib (Ubuntu Jammy):
status: New → In Progress
assignee: Skipper Bug Screeners (skipper-screen-team) → Brian Murray (brian-murray)
Revision history for this message
Frank Heimes (fheimes) wrote :

SRU justification provides and added to the bug description.

description: updated
Frank Heimes (fheimes)
description: updated
Revision history for this message
Ilya Leoshkevich (iii-i) wrote :

We just realized that we had DFLTCC support in 20.04, so we need to put this fix there as well.

Frank Heimes (fheimes)
description: updated
Revision history for this message
Frank Heimes (fheimes) wrote :

Let me check if the patch applies to the focal version, too.
A separate debdiff is needed for focal anyway.

Revision history for this message
Frank Heimes (fheimes) wrote :

The patch applies cleanly not only to the impish and jammy versions, but also to the focal version of zlib.

To be on the save side I did another test build for focal:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1961427

And here is the focal debdiff attached (debdiff_zlib_1.2.11.dfsg-2ubuntu1.2_to_zlib_1.2.11.dfsg-2ubuntu1.3).

Revision history for this message
Frank Heimes (fheimes) wrote :

And - just for the reason of completeness - the debdiff for impish, that is equal to the jammy debdiff, except the release name at the top line in the changelog (debdiff_zlib_1.2.11.dfsg-2ubuntu7_to_zlib_1.2.11.dfsg-2ubuntu8).

Revision history for this message
Frank Heimes (fheimes) wrote :

Sorry, the impish version needs to be of course "1.2.11.dfsg-2ubuntu7.1" - to keep the upgrade path working.
Here is the updated debdiff (debdiff_zlib_1.2.11.dfsg-2ubuntu7_to_zlib_1.2.11.dfsg-2ubuntu7.1_impish.diff).

Revision history for this message
bugproxy (bugproxy) wrote : debdiff_zlib_1.2.11.dfsg-2ubuntu7_to_zlib_1.2.11.dfsg-2ubuntu8

Default Comment by Bridge

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2022-03-15 18:41 EDT-------
zlib1g s390x 1:1.2.11.dfsg-2ubuntu1.3 passes all of my tests on focal, thanks!

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Currently dropping 1:1.2.11.dfsg-2ubuntu8 from jammy-proposed per the uploader's request:

09:55 < fheimes> Dear ubuntu-release team, could you please drop/remove zlib 2.11.dfsg-2ubuntu8 from jammy-proposed (LP#1961427)?
09:55 < fheimes> An incompatibility with HTSlib was detected, I'm in contact with the initial reporter and author of the patch to get that solved, but this might take a while (hence I restart from scratch on this is probably best).

Frank Heimes (fheimes)
Changed in zlib (Ubuntu Jammy):
status: In Progress → Incomplete
Changed in ubuntu-z-systems:
status: New → Incomplete
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-05-04 05:20 EDT-------
It seems there is no progress with this issue for a while. What is next? Is the reason for the incompatibility found meanwhile?

Revision history for this message
Frank Heimes (fheimes) wrote :

We are waiting on Ilya (iii-i) for a refined patch.
He mentioned that this can take a while ...

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-05-04 08:51 EDT-------
Good that this was pinged - it seems that we hit a deadlock here :-)

I was waiting on Ubuntu to bump the HTSlib version (or cherry-pick https://github.com/samtools/htslib/pull/1258), where the incompatibility issue is resolved. The original zlib patch should then just work.

Revision history for this message
Frank Heimes (fheimes) wrote :

Good to have this follow-up, indeed ...
(For a fix in htslib either a new LP bug is needed or htslib needs to be added to this one as 'affecting package'.)

So I did some investigations and found that the "Remove compressBound assertions." merge request #1258 got upstream accepted with htslib v1.13:
$ git log --oneline --grep "Remove compressBound assertions"
515f6df Remove compressBound assertions. (PR #1258)
$ git tag --contains 515f6df
1.13
1.14
1.15
1.15.1
But we already have v1.13 in the latest Ubuntu releases:
$ rmadison -a source htslib
 htslib | 0.2.0~rc3-1 | trusty/universe | source
 htslib | 1.2.1-2ubuntu1 | xenial/universe | source
 htslib | 1.7-2 | bionic/universe | source
 htslib | 1.10.2-3 | focal/universe | source
 htslib | 1.13+ds-2 | impish/universe | source
 htslib | 1.13+ds-2build1 | jammy/universe | source
 htslib | 1.13+ds-2build1 | kinetic/universe | source

So PR #1258 is already included in the htslib version of kinetic/22.10, jammy/22.04 and impish/21.10 (and I even double-checked the code).
It's not included in focals/20.04 v1.10.x - and it looks like the PR cannot be simply applied to v1.10, since the code in v1.10 is too different - which will be a problem.

Since the regression was found in jammy's zlib 1.2.11.dfsg-2ubuntu8 in combination with 1.13+ds-2build1 - I am unsure if PR #1258 is really the fix or if there something else needed on top?
Any thoughts?

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-05-06 06:22 EDT-------
Here are the originally reported failures:

> But I found in "bedtools":
> bedtools: bgzf.c:351: bgzf_hopen: Assertion
> `compressBound(BGZF_BLOCK_SIZE) < BGZF_MAX_BLOCK_SIZE' failed. test-
> intersect.sh: line 201: 1981 Aborted (core dumped)
> $htsutil samtobam one_block.sam one_block.bam
>
> or in libbio:
> perl: bgzf.c:142: bgzf_open: Assertion
> `compressBound(BGZF_BLOCK_SIZE) < BGZF_MAX_BLOCK_SIZE' failed.Failed
> 268/268 subtests

I just double-checked, and apparently bedtools bundles htslib (under src/utils/htslib). I didn't check libbio, but most likely it does this as well. So the patch must be cherry-picked into all of the bundled copies.

Revision history for this message
Frank Heimes (fheimes) wrote (last edit ):

That bundled htslib seems to make the situation a bit messy,
since bedtools does not clearly list the htslib version that was bundled or I can't find it (oh, maybe it's 1.9) and the commit does not apply cleanly.
So my guess is that this should be fixed in bedtools upstream first... but even that is not perfect, since it would fix the latest upstream version only.
Well, this needs some further discussions ...

Revision history for this message
Frank Heimes (fheimes) wrote :

So, I've now created an updated bedtools package for jammy (having patched the bundled htslib with 515f6df-Remove-compressBound-assertions.-PR-1258.patch) for jammy,
and since the zlib package received some CVE patches meanwhile, I also created a new patched zlib version (that incl. 410-lp1961427.patch) for jammy.

@Ilya would you mind trying this combination on a z15 running jammy?
Meanwhile I will also look at the stand-alone htslib (and wonder how often htslib might be bundled somewhere else too ...)

Once I get your okay, I will restart this work, by patching kinetic first and then going down the road ... (and reaching out to the maintainer of the Debian packages, too).

Revision history for this message
bugproxy (bugproxy) wrote :

Well, that bundled htslib seems to make the situation a bit messy,
So my guess is that this should be fixed in bedtools upstream first...

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-05-10 07:14 EDT-------
Hi, thanks for looking into this. I'll wait until https://launchpad.net/~fheimes/+archive/ubuntu/lp1961427 finishes the build and try it.

In the meantime I've asked the bedtools upstream to cherry-pick the htslib commit: https://github.com/arq5x/bedtools2/pull/993.

Revision history for this message
Frank Heimes (fheimes) wrote :

Ok, thx.
(and yes, the builders are currently a bit busy - just noticed the 140 jobs in the queue ...
guess it's a matter of maybe 2+ hours ...)

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Usually we try to avoid using embedded copies of code. Is it at all possible to convert some of the affected packages to use/reuse a shared library instead?

Revision history for this message
Frank Heimes (fheimes) wrote :

@xnox Well, initially I was also a bit puzzled noticing that not the standalone htslib is used, but an embedded htslib in bedtools.
But this is also upstream the case and htslib is not the only component that is bundled:
$ ls ./src/utils/
BamTools BlockedIntervals fileType lineFileUtilities tabFile
BamTools-Ancillary chromsweep general NewChromsweep ToolBase
bedFile Contexts GenomeFile Point VectorOps
bedFilePE driver gzstream RecordOutputMgr version
bedGraphFile Fasta htslib sequenceUtilities
BinTree FileRecordTools KeyListOps stringUtilities
And solving the situation for all of these will be a nighmare with a high risk of introducing problems on top and more difficulties on maintaining and making sure that the components are in place in the correct version ...

Revision history for this message
Frank Heimes (fheimes) wrote :

Just for the records: the standalone htslib package 1.13+ds-2build1 in jammy includes PR-1258 already.
(and the PPA build is finished)

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-05-16 06:17 EDT-------
With the new combination, bedtools testsuite is green and zlib passes all of my tests too. So it looks good, thanks!

Revision history for this message
Frank Heimes (fheimes) wrote (last edit ):

Thx Ilya, restarted to work on this by adding updated bedtools to the PPA ...
Interestingly bedtools version 2.27.1+dfsg from focal does not have htslib bundled:
$ ls -l src/utils/h*
ls: cannot access 'src/utils/h*': No such file or directory
$ ls src/utils/
BamTools bedGraphFile Contexts fileType KeyListOps RecordOutputMgr ToolBase
BamTools-Ancillary BinTree driver general lineFileUtilities sequenceUtilities VectorOps
bedFile BlockedIntervals Fasta GenomeFile NewChromsweep stringUtilities version
bedFilePE chromsweep FileRecordTools gzstream Point tabFile
and I cannot find any reference to htslib in that particular package version.

Ilya, on which Ubuntu release did you test (LP comment #26)?
I assume it was not 20.04, but probably jammy/22.04?
So would you mind testing the patched zlib package 1.2.11.dfsg-2ubuntu1.3 again in focal ?
Things are definitely different these and with the missing reference to htslib I think there is a chance that it might work there w/o touching bedtools. - Thx

Revision history for this message
Frank Heimes (fheimes) wrote :
description: updated
Changed in htslib (Ubuntu Impish):
status: New → Invalid
Changed in htslib (Ubuntu Jammy):
status: New → Invalid
Changed in bedtools (Ubuntu Focal):
status: New → Invalid
Revision history for this message
Frank Heimes (fheimes) wrote :
Revision history for this message
Frank Heimes (fheimes) wrote :
Revision history for this message
Frank Heimes (fheimes) wrote :
Revision history for this message
Frank Heimes (fheimes) wrote (last edit ):

I would like to proceed on this (to not loose to much time...)

To sum things up:

On top of the actual zlib fix,
there is another patch 'Remove compressBound assertions. (PR #1258)'
needed for htslib.
Unfortunately there is a standalone 'htslib' package,
as well as a htslib version included in (some) 'bedtools' packages.
Both need to be patched.
The 'htslib' version '1.13+ds' (as it is part of I, J and K),
already includes the patch,
hence only htslib '1.10.2' in focal needs to be patched.
The bedtools version '2.30.0+dfsg' (as it is part of I, J and K),
requires the patch,
but version '2.27.1+dfsg' bedtools in focal does not incl. and embedded
htslib, hence does not need to be (actually can't be) patched.

Changed in htslib (Ubuntu):
status: New → Invalid
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-06-15 14:00 EDT-------
I don't really remember, so I retested.

20.04: 1.2.11.dfsg-2ubuntu1.3 fixes the problem (I had to install the .deb file manually with dpkg).
22.04: 1.2.11.dfsg-2ubuntu9.1 fixes the problem.

Revision history for this message
Frank Heimes (fheimes) wrote (last edit ):

Thx for the retest!
With that I would go from Incomplete back to In Progress ...

Changed in ubuntu-z-systems:
status: Incomplete → In Progress
Changed in zlib (Ubuntu):
status: Incomplete → In Progress
Revision history for this message
Simon Chopin (schopin) wrote : Re: [Bug 1961427] Re: zlib: compressBound() returns an incorrect result on z15

I'm just watching from the sidelines here, but why do we need to have
a transition for this? Is there an ABI break I'm not aware of? Or is zlib
usually statically linked in Debian packages?

Revision history for this message
Frank Heimes (fheimes) wrote (last edit ):

You are right, these are 'just' fixes w/o ABI changes and I am not aware (couldn't find) of any cases of static linkages - I apologize.
So fortunately the complexity of this will be far less.

Revision history for this message
Frank Heimes (fheimes) wrote :

Since Ubuntu 21.10 (Impish Indri) will soon reach its End of Life on July 14 2022,
this will not land in impish anymore, hence I mark the impish entries as 'Won't Fix'.

Changed in bedtools (Ubuntu Impish):
status: New → Won't Fix
Changed in htslib (Ubuntu Impish):
status: Invalid → Won't Fix
Changed in zlib (Ubuntu Impish):
status: New → Won't Fix
Changed in zlib (Ubuntu Jammy):
status: Incomplete → New
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2022-07-06 20:34 EDT-------
Changing status to: REJECTED - WILL_NOT_FIX for this 21.10 bug
as we will reach end of life of Ubuntu 21.10 (impish) in one week from now

Revision history for this message
Frank Heimes (fheimes) wrote :

Just to clarify - only 21.10/impish got dropped.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-07-07 07:55 EDT-------
(In reply to comment #13)
> We just realized that we had DFLTCC support in 20.04, so we need to put this
> fix there as well.

So this still gets fixed in Focal (20.04).

bugproxy (bugproxy)
tags: added: targetmilestone-inin20045
removed: targetmilestone-inin2110
Revision history for this message
bugproxy (bugproxy) wrote :

Due to the fact that now in total 3 packages are involved and two libraries (zlib and htslib, well and the libraries included in bedtools), this will cause a bigger transition (especially for htslib in focal, and zlib itself on all affected Ubuntu releases).
(The transition(s) need to be planned here: https://people.canonical.com/~ubuntu-archive/transitions/index.html)

Revision history for this message
bugproxy (bugproxy) wrote : reverse dependencies, gathered on focal.txt

Default Comment by Bridge

Revision history for this message
Frank Heimes (fheimes) wrote (last edit ):

zlib debdiffs for kinetic, jammy and focal (latest, with proper dep3)

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

Since the zlib package progressed meanwhile due to a security update, I created new debdiffs for jammy and kinetic - see debdiffs_J_K.tar.gz.

Revision history for this message
bugproxy (bugproxy) wrote : zlib debdiffs for jammy and kinetic (latest)

Default Comment by Bridge

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2022-07-21 05:51 EDT-------
Comment on attachment 153124
zlib_1.2.11.dfsg-2ubuntu7_to_zlib_1.2.11.dfsg-2ubuntu8 (jammy)

this is obsolete - has been replaced by newer version

------- Comment From <email address hidden> 2022-07-21 05:52 EDT-------
Comment on attachment 154585
reverse dependencies, gathered on focal.txt

obsolete - has been removed

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package bedtools - 2.30.0+dfsg-2ubuntu1

---------------
bedtools (2.30.0+dfsg-2ubuntu1) kinetic; urgency=medium

  * Add d/p/515f6df-Remove-compressBound-assertions-PR-1258.patch in support
    of fixing 'zlib: compressBound() returns an incorrect result on z15'.
    (LP: #1961427)

 -- Frank Heimes <email address hidden> Thu, 19 May 2022 09:52:42 +0200

Changed in bedtools (Ubuntu):
status: New → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote :

I believe I've sponsored all the uploads necessary to see this bug fixed.

Changed in bedtools (Ubuntu Jammy):
assignee: nobody → Brian Murray (brian-murray)
Changed in htslib (Ubuntu Focal):
assignee: nobody → Brian Murray (brian-murray)
Changed in zlib (Ubuntu Focal):
assignee: nobody → Brian Murray (brian-murray)
Revision history for this message
Frank Heimes (fheimes) wrote :

I've noticed that the updated zlib for kinetic just landed in -proposed:
"zlib | 1:1.2.11.dfsg-2ubuntu10 | kinetic-proposed"
hence I'm updating the corresponding entry to Fix Committed.

Changed in zlib (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Frank Heimes (fheimes) wrote (last edit ):

zlib 1:1.2.11.dfsg-2ubuntu10 is stuck in kinetic-proposed for some time:

http://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#zlib https://wiki.ubuntu.com/ProposedMigration

Several of the initial regressions were due to timeouts and network issues and are solved meanwhile. Two (red) regressions are left (aot):

I opened LP#1983128 for:
"autopkgtest for linux-lowlatency/5.19.0-1001.1: amd64: Pass, arm64: Regression ♻"

and due to:
"autopkgtest for libbio-samtools-perl/1.43-3build2: amd64: Pass, arm64: Pass, armhf: Pass, ppc64el: Pass, s390x: Regression ♻"
I re-ran the autopkgtest (using LXD) for libbio-samtools-perl explicitly on s390x (using the zlib PPA) which worked fine for me (see attachemnt),
hence I'm going to ask for a re-run. (I don't have permissions yet to re-trigger it myself.)

(While looking into libbio-samtools-perl and other related packages, I found that there is yet another htslib bundled in package tabix - which is not great. But this had at least the needed patch already included.)

Revision history for this message
Frank Heimes (fheimes) wrote :

Looks like libbam-dev is the reason for the libbio-samtools-perl regression
on s390x. libbam-dev is used by libbio-samtools-perl and build based on it's
source package samtools-legacy, which includes another version of bgzf,
that misses the 'Remove compressBound assertions' (PR #1258) fix.

Problematic is that the code differs quite a bit from the one in the htslib
or bedtools package (no bgzf_hopen etc. - looks pretty much out of date),
so the PR does not apply cleanly.

According to upstream 'https://github.com/lh3/samtools-legacy' the project
is "For testing only." and marked as "DON'T USE!" and it's last update
is from Nov 9, 2013.
I've created a separate 'update-excuse' but for this: LP#1983255

Revision history for this message
bugproxy (bugproxy) wrote :

(While looking into libbio-samtools-perl and other related packages, I found that there is yet another htslib bundled in tabix - which is not great. But this at at least already the needed patch.)

Revision history for this message
Frank Heimes (fheimes) wrote :

There is now a solution for LP#1983255.

Meanwhile I've noticed that gdcm is now flagged as a new regression for zlib's proposed miration at 'update excuses' (was not the case before).

Found that this happened on Debian too:
https://<email address hidden>/msg1863513.html
(with exactly the same error: https://ci.debian.net/data/autopkgtest/testing/s390x/g/gdcm/23826805/log.gz)
but seems to be solved meanwhile: https://qa.debian.org/excuses.php?package=gdcm
So can obviously be solved by an update from (Ubuntu) 3.0.13-2 to (Debian) 3.0.13-3 (sync),
( according to changelog: https://metadata.ftp-master.debian.org/changelogs//main/g/gdcm/gdcm_3.0.13-3_changelog )

Just noticed that it was already synched a few hours ago:
"3.0.13-3 release (universe) 19 hours ago"
(https://launchpad.net/ubuntu/+source/gdcm)

So I assume it's a matter of time until update-excuses reflects this gdcm update and get's again rid of the regression.

Revision history for this message
Brian Murray (brian-murray) wrote :

Actually, somebody will need to manually retrigger the gdcm test with the version of zlib in -proposed and I'll do that now.

Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted htslib into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/htslib/1.10.2-3ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in htslib (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Steve Langasek (vorlon) wrote :

Hello bugproxy, or anyone else affected,

Accepted zlib into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zlib/1:1.2.11.dfsg-2ubuntu1.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in zlib (Ubuntu Focal):
status: New → Fix Committed
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (zlib/1:1.2.11.dfsg-2ubuntu1.4)

All autopkgtests for the newly accepted zlib (1:1.2.11.dfsg-2ubuntu1.4) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

libtabixpp/1.1.0-3 (s390x)
linux-hwe-5.11/5.11.0-61.61 (armhf)
smalt/0.7.6-9 (s390x)
linux-hwe-5.15/5.15.0-45.48~20.04.1 (arm64, armhf)
rtags/2.37-1 (amd64)
htslib/1.10.2-3 (s390x)
linux-hwe-5.13/5.13.0-52.59~20.04.1 (armhf)
bacula/9.4.2-2ubuntu5 (amd64)
ruby2.7/2.7.0-5ubuntu1.7 (arm64)
samtools/1.10-3 (amd64)
libbio-samtools-perl/1.43-2build1 (s390x)
vcftools/0.1.16-1build1 (s390x)
kallisto/0.46.1+dfsg-2build1 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#zlib

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Frank Heimes (fheimes) wrote :

Regarding zlib focal-proposed migration:

For libbio-samtools-perl it's the same situation like on kinetic.
samtools-legacy requires the PR #1258, and libbio-samtools-perl a rebuild (to link against the new libbio-dev). See update-excuse bug LP#1983255.

On top vcftools misses the 'Remove compressBound assertions' (PR #1258) fix as well.
I created a separate update-execuse bug LP#1983867.

The mentioned issue with htslib/1.10.2-3 is because the patched htslib 1.10.2-3ubuntu0.1 (that again incl. PR #1258) is needed:
https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#htslib

This htslib 1.10.2-3ubuntu0.1 will at least solve kallisto, libtabixpp and probably samtools as well.

The isues with the kernels linux-hwe-5.11, linux-hwe-5.13 and linux-hwe-5.15 are all time outs on arm64, resspectively amrhf builds. So it's worth to retrigger these.

Revision history for this message
Lukas Märdian (slyon) wrote :

The zlib patch is already waiting in Jammy's unapproved queue, so I'm marking it "In Progress".

Changed in zlib (Ubuntu Jammy):
status: New → In Progress
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted bedtools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/bedtools/2.30.0+dfsg-2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in bedtools (Ubuntu Jammy):
status: New → Fix Committed
tags: added: verification-needed-jammy
Revision history for this message
Steve Langasek (vorlon) wrote :

Hello bugproxy, or anyone else affected,

Accepted zlib into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zlib/1:1.2.11.dfsg-2ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in zlib (Ubuntu Jammy):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (zlib/1:1.2.11.dfsg-2ubuntu9.1)

All autopkgtests for the newly accepted zlib (1:1.2.11.dfsg-2ubuntu9.1) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

sbcl/2:2.1.11-1 (amd64, arm64)
openjdk-lts/11.0.16+8-0ubuntu1~22.04 (armhf)
booth/1.0-237-gdd88847-4ubuntu2.2 (s390x)
mysql-8.0/8.0.30-0ubuntu0.22.04.1 (amd64)
cluster-glue/1.0.12-20ubuntu3 (ppc64el)
rspamd/2.7-1build2 (amd64)
libbio-samtools-perl/1.43-3build2 (s390x)
casync/2+20201210-1build1 (ppc64el)
bedtools/2.30.0+dfsg-2 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#zlib

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zlib - 1:1.2.11.dfsg-2ubuntu10

---------------
zlib (1:1.2.11.dfsg-2ubuntu10) kinetic; urgency=medium

  * d/p/410-lp1961427.patch ported from zlib #410, fixing
    compressBound() with hw acceleration. LP: #1961427
    Thanks to Ilya Leoshkevich <email address hidden>.
    In addition a patch is needed for bedtools.

 -- Frank Heimes <email address hidden> Thu, 21 Jul 2022 09:30:05 +0100

Changed in zlib (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2022-09-05 11:31 EDT-------
zlib1g-dev s390x 1:1.2.11.dfsg-2ubuntu9.1 and zlib1g s390x 1:1.2.11.dfsg-2ubuntu9.1 pass all of my tests.

Revision history for this message
Frank Heimes (fheimes) wrote (last edit ):

Thx Iliya! (updating tag accordingly)

tags: added: verification-done verification-done-focal verification-done-jammy
removed: verification-needed verification-needed-focal verification-needed-jammy
tags: added: foundations-todo
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zlib - 1:1.2.11.dfsg-2ubuntu9.1

---------------
zlib (1:1.2.11.dfsg-2ubuntu9.1) jammy; urgency=medium

  * d/p/410-lp1961427.patch ported from zlib #410, fixing
    compressBound() with hw acceleration. LP: #1961427
    Thanks to Ilya Leoshkevich <email address hidden>.
    In addition a patch is needed for bedtools.

 -- Frank Heimes <email address hidden> Thu, 21 Jul 2022 10:30:05 +0100

Changed in zlib (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for zlib has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package bedtools - 2.30.0+dfsg-2ubuntu0.1

---------------
bedtools (2.30.0+dfsg-2ubuntu0.1) jammy; urgency=medium

  * Add d/p/515f6df-Remove-compressBound-assertions-PR-1258.patch in support
    of fixing 'zlib: compressBound() returns an incorrect result on z15'.
    (LP: #1961427)

 -- Frank Heimes <email address hidden> Thu, 19 May 2022 09:52:42 +0200

Changed in bedtools (Ubuntu Jammy):
status: Fix Committed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hey Frank! You switched the focal verification tags, but I do not see any verification of the focal packages. Comment #65 only mentions versions for jammy. Have the focal packages been tested in any way?

Switching back tags for focal.

tags: added: verification-needed verification-needed-focal
removed: verification-done verification-done-focal
bugproxy (bugproxy)
tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Frank Heimes (fheimes) wrote :

Well, I had comment #11 in mind where the 1:1.2.11.dfsg-2ubuntu1.3 was successfully verified on focal.
Meanwhile that version got overruled by a security update:
zlib (1:1.2.11.dfsg-2ubuntu1.3) focal-security; urgency=medium
and became 'already used', hence a new version was needed for this bug and so we are now at:
1:1.2.11.dfsg-2ubuntu1.4 - but it incl. the same patches.
Anyway, I'll ask to get 1:1.2.11.dfsg-2ubuntu1.4 verified again on focal.

bugproxy (bugproxy)
tags: added: verification-needed verification-needed-focal
removed: verification-done verification-done-focal
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2022-09-22 08:10 EDT-------
zlib1g s390x 1:1.2.11.dfsg-2ubuntu1.4 passes all of my tests.

------- Comment From <email address hidden> 2022-09-22 08:18 EDT-------
@Ilya: thanks a lot for your instant verification on focal!

With that, I am updating the tags / keywords to: verification-done, verification-done-focal, verification-done-jammy

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
tags: removed: foundations-todo
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zlib - 1:1.2.11.dfsg-2ubuntu1.4

---------------
zlib (1:1.2.11.dfsg-2ubuntu1.4) focal; urgency=medium

  * d/p/410-lp1961427.patch ported from zlib #410, fixing
    compressBound() with hw acceleration. LP: #1961427
    Thanks to Ilya Leoshkevich <email address hidden>.
    In addition a patch is needed in htslib.

 -- Frank Heimes <email address hidden> Thu, 21 Jul 2022 10:30:05 +0100

Changed in zlib (Ubuntu Focal):
status: Fix Committed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package htslib - 1.10.2-3ubuntu0.1

---------------
htslib (1.10.2-3ubuntu0.1) focal; urgency=medium

  * Add d/p/515f6df-Remove-compressBound-assertions-PR-1258.patch in support
    of fixing 'zlib: compressBound() returns an incorrect result on z15'.
    Due a fuzz 2 a minor adjustment of the context was needed.
    (LP: #1961427)

 -- Frank Heimes <email address hidden> Wed, 15 Jun 2022 16:45:11 +0200

Changed in htslib (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.