Ubuntu archive is missing SHA-1/SHA-256 hashes for some packages

 subscribe ubuntu-archive

We still use apt-ftparchive for the Ubuntu primary archive, which only includes SHA1 and SHA256 hashes if the .dsc does. The affected packages are all quite old as expected, except for language packs, which suggests a bug in the language pack builder.

I just ran in to this in our new mirroring system for pulling packages, hence my lateness to the party.

A cursory glance at my list suggests that William's characterization is correct; except for the language packs, these are pretty old packages.

There is an annoying wrinkle that .dsc's are normally gpg-signed by the package maintainer, so the .dsc could only be changed by removing the signature. Perhaps that's the right thing; I'm not sure.

However, nothing requires the Sources file to have only the headers in the .dsc; it would at least be an improvement to have the Sources file contain full hashes when it's generated, even if the .dsc (or the .changes) lack them.

By the way, this is not a low importance issue in my opinion. This is tantamount to having no hash protection at all; it is possible with inexpensive hardware to force an md5 collision in a few hours.

An internal audit of the entire mirror is being run with http://paste.ubuntu.com/1521026/

Now, due to our general distaste for changing releases post-release, I doubt we'll want to fix this in stable releases (though, fixing langpack-o-matic to get this right on the next SRU round of langpacks and going forward is something I'll look into).

Doing something about sources that haven't been uploaded since hardy might be a good idea, though.

And, to be fair, while I do agree that this is an issue, I'm not sure what the practical impact of someone forcing an md5 collision on a source package is. Now, if we also have this issue for some binary packages, that's a much bigger deal.

For people who don't have a full mirror to play with, results of the above audit script are available at http://people.canonical.com/~ubuntu-archive/checksum-audit/

I haven't correlated them into anything particularly sane yet, but it does indeed look like the scope of the damage is "langpacks suck and need fixing" and "some package haven't been updated since hardy, wow".

Having the Sources file have better hashes than md5 doesn't improve things for now, as "apt-get source" only checks for md5.

Huhuh? That's sad.

But apt-get source isn't the only tool. ;)

On Fri, Jan 11, 2013 at 2:18 PM, Marc Deslauriers <
<email address hidden>> wrote:

> Having the Sources file have better hashes than md5 doesn't improve
> things for now, as "apt-get source" only checks for md5.
>
> --
> You received this bug notification because you are a member of Goobuntu
> Team, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1078697
>
> Title:
> Ubuntu archive is missing SHA-1/SHA-256 hashes for some packages
>
> Status in Launchpad itself:
> Triaged
>
> Bug description:
> As part of the Debian derivatives census, we are doing some checks on
> all derivatives. We noticed that a number of source packages are missing
> SHA-1/SHA-256 hashes. You may have inherited this issue from Debian, we
> had the same issue until recently. Here are some sample messages from
> the report below, which is generated daily.
>
> WARNING: source cvstrac 2.0.1-3: SHA-256 hashes but no hash for the dsc
> file
> WARNING: source cvstrac 2.0.1-3: SHA-1 hashes but no hash for the dsc
> file
> WARNING: source diveintopython 5.4-2ubuntu2: no SHA-256 hash
> WARNING: source diveintopython 5.4-2ubuntu2: no SHA-1 hash
>
> http://dex.alioth.debian.org/census/Ubuntu/check-package-list
>
> Please ignore the warnings about GPG and InRelease stuff, they are due
> to python-apt not supporting some things in Debian squeeze.
>
> affects launchpad
> subscribe ubuntu-archive
>
> --
> bye,
> pabs
>
> http://wiki.debian.org/PaulWise
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/launchpad/+bug/1078697/+subscriptions
>

This needs fixing in apt-ftparchive before Launchpad can do anything.

Also, MD5 collisions aren't hugely concerning here. It's a preimage that would be more of a problem, and there's no serious preimage attack known on MD5 today. I agree that this isn't a good situation, but it's not "everything is broken with a few hours of computation" bad.

I'm not so sure. It's true that the known attacks are collision attacks.
Yet, collision attacks can be used to mount data-integrity attacks that
replace specific files in archives, at least, with the trick at
http://eprint.iacr.org/2004/356.pdf. That depends on having the colliding
blocks happen to have enough bytes in them that the choice of colliding
block functions as a jump table for a self-extracting archive. It obviously
doesn't directly map to a .dsc.

Generally people regard it as dead as soon as collisions are *found *even
if nobody knows how to generate them. (Note that in the paper I cite above,
they simply used the colliding blocks published by the original Chinese
researchers, who at the time had not made public their strategy for finding
collisions.) Someone interested in pwning md5 could well have built an
extremely large library of colliding blocks by now, including with ones
that look like tar headers and the like. I'm just not so confident.

On Fri, Jan 11, 2013 at 3:22 PM, William Grant <email address hidden>wrote:

> This needs fixing in apt-ftparchive before Launchpad can do anything.
>
> Also, MD5 collisions aren't hugely concerning here. It's a preimage that
> would be more of a problem, and there's no serious preimage attack known
> on MD5 today. I agree that this isn't a good situation, but it's not
> "everything is broken with a few hours of computation" bad.
>
> ** Also affects: apt (Ubuntu)
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are a member of Goobuntu
> Team, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1078697
>
> Title:
> Ubuntu archive is missing SHA-1/SHA-256 hashes for some packages
>
> Status in Launchpad itself:
> Triaged
> Status in “apt” package in Ubuntu:
> New
>
> Bug description:
> As part of the Debian derivatives census, we are doing some checks on
> all derivatives. We noticed that a number of source packages are missing
> SHA-1/SHA-256 hashes. You may have inherited this issue from Debian, we
> had the same issue until recently. Here are some sample messages from
> the report below, which is generated daily.
>
> WARNING: source cvstrac 2.0.1-3: SHA-256 hashes but no hash for the dsc
> file
> WARNING: source cvstrac 2.0.1-3: SHA-1 hashes but no hash for the dsc
> file
> WARNING: source diveintopython 5.4-2ubuntu2: no SHA-256 hash
> WARNING: source diveintopython 5.4-2ubuntu2: no SHA-1 hash
>
> http://dex.alioth.debian.org/census/Ubuntu/check-package-list
>
> Please ignore the warnings about GPG and InRelease stuff, they are due
> to python-apt not supporting some things in Debian squeeze.
>
> affects launchpad
> subscribe ubuntu-archive
>
> --
> bye,
> pabs
>
> http://wiki.debian.org/PaulWise
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/launchpad/+bug/1078697/+subscriptions
>

It certainly should be regarded as entirely broken, but it's not world-burningly critical to fix the old releases.

Yeah, I wouldn't worry about old releases, just the current ones (but that
does include lucid, precise, and quantal, and oneiric unless it takes a
while to sorto ut).

On Fri, Jan 11, 2013 at 3:58 PM, William Grant <email address hidden>wrote:

> It certainly should be regarded as entirely broken, but it's not world-
> burningly critical to fix the old releases.
>
> --
> You received this bug notification because you are a member of Goobuntu
> Team, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1078697
>
> Title:
> Ubuntu archive is missing SHA-1/SHA-256 hashes for some packages
>
> Status in Launchpad itself:
> Triaged
> Status in “apt” package in Ubuntu:
> New
>
> Bug description:
> As part of the Debian derivatives census, we are doing some checks on
> all derivatives. We noticed that a number of source packages are missing
> SHA-1/SHA-256 hashes. You may have inherited this issue from Debian, we
> had the same issue until recently. Here are some sample messages from
> the report below, which is generated daily.
>
> WARNING: source cvstrac 2.0.1-3: SHA-256 hashes but no hash for the dsc
> file
> WARNING: source cvstrac 2.0.1-3: SHA-1 hashes but no hash for the dsc
> file
> WARNING: source diveintopython 5.4-2ubuntu2: no SHA-256 hash
> WARNING: source diveintopython 5.4-2ubuntu2: no SHA-1 hash
>
> http://dex.alioth.debian.org/census/Ubuntu/check-package-list
>
> Please ignore the warnings about GPG and InRelease stuff, they are due
> to python-apt not supporting some things in Debian squeeze.
>
> affects launchpad
> subscribe ubuntu-archive
>
> --
> bye,
> pabs
>
> http://wiki.debian.org/PaulWise
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/launchpad/+bug/1078697/+subscriptions
>

I suspect when William said "old releases" he meant "already-published releases for which we generally and strongly prefer not to regenerate indices". That said, a one-time regen of only Sources and no other indices, might not be the worst thing ever, if we fix apt-ftparchive to include the missing bits. I'd still prefer not to.

That said, under no circumstances will we be "fixing" the .dsc (or replacing the source packages) in stable releases.

If you wait a bit longer the fix for apt-ftparchive is 3 years old: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=567343
That is rev 1875.1.95 in bzr and what pabs refers to as until recently (minus the time needed to get this onto ftp-master box of course) as far as I know.

And of course @mdeslaur, apt-get source does more than just checking MD5. It does what it does for all other downloads as well: Take the "best" checksum it knows and is available for checking if it isn't forced to use another (Acquire::ForceHash). What it does do with MD5 only is checking if the file on the disc matches the file we would download and if it does skipping the download as already done, which should be fixed (so that we can drop MD5 at some point) but has no real security implications as someone with write access to your local disk in that directory has better things to do …

Indeed, that change seems to do what is desired. But 7 months later, in September 2010, Ubuntu bug #633967 was fixed, making apt-ftparchive only include hashes if they're in the dsc: <http://bazaar.launchpad.net/~ubuntu-branches/debian/sid/apt/sid/revision/21#ftparchive/writer.cc>. This change was backported to Lucid in 0.7.25.3ubuntu9.2, resulting in the behaviour we see on production today.

I suspect that Debian somewhat accidentally resolved the problem by dropping apt-ftparchive and moving to database-backed index generation.

This commit should prevent apt-ftparchive to generate Checksum-listings which just includes the dsc file and nothing else (General mode of operation: It copies the Checksum-listings from the dsc file and adds the dsc file to it). So that would explain the diveintopython example (if diveintopython.dsc really has no SHA1/SHA256 checksums included), but not cvstrac as cvstrac.dsc seems to include checksums apt-ftparchive copied but "forgot" to add checksums for the dsc file.

Unlike Debian, Ubuntu's post-release updates go to a separate pocket -- the release pocket is frozen and its indices are never regenerated after release. Production was upgraded from hardy to lucid between lucid and maverick, so we'd expect lucid's Sources to omit the .dsc, while maverick's would include it. Comparing the cvstrac stanzas in http://releases.ubuntu.com/ubuntu/dists/lucid/universe/source/Sources.gz and http://old-releases.ubuntu.com/ubuntu/dists/maverick/universe/source/Sources.gz would seem to confirm this.

(To clarify, the hardy->lucid upgrade is important because it pulled in apt-ftparchive >= 0.7.25.3 on ftpmaster.)

@David, please see my response to comment #15 in bug #1098738

All the langpacks that just landed in precise-proposed should have a full compliment of hashes in their .dsc files, FWIW.

Here's a first stab at making apt-ftparchive generate the missing hashes. Review appreciated.

@Marc, looking solely at the patch, it looks good to me. (I haven't looked at the rest of the apt code to make sure it fits.)

On Thu, Jan 31, 2013 at 01:00:54PM -0000, Marc Deslauriers wrote:
> Here's a first stab at making apt-ftparchive generate the missing
> hashes. Review appreciated.
>
> ** Patch added: "apt_0.9.7.7ubuntu2~md1.debdiff"
> https://bugs.launchpad.net/launchpad/+bug/1078697/+attachment/3508960/+files/apt_0.9.7.7ubuntu2%7Emd1.debdiff

This looks good, thanks a lot!

I put this to the lp:~mvo/apt/add-missing-dsc-hashes/ branch and added
a small integration test (attached for review as well).

Cheers,
 Michael

I assume we need a precise version of this branch too?

Fixes for Raring and Precise have been uploaded, awaiting approval.

mvo provided a patch against Lucid's version (see attached branch) and I tested this against lucid-cat. It seemed to work so I uploaded to lucid-cat. Caveat: lucid-cat version doesn't produce sha512 checksums (only sha1 and sha256).

IS has been notified, so hopefully we'll see this land on Launchpad in production soon.

This bug was fixed in the package apt - 0.9.7.7ubuntu4

---------------
apt (0.9.7.7ubuntu4) raring; urgency=low

  [ Michael Vogt ]
  * test/integration/test-bug-1078697-missing-source-hashes:
    - add test for deb-src hash generation

  [ Marc Deslauriers ]
  * make apt-ftparchive generate missing deb-src hashes (LP: #1078697)
 -- Michael Vogt <email address hidden> Thu, 11 Apr 2013 14:52:15 +0200

Still waiting for the SRU into Precise, but the package has been accepted into lucid-cat and should land on Launchpad at its next update, currently scheduled for next Wednesday.

Hello Paul, or anyone else affected,

Accepted apt into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/0.8.16~exp12ubuntu10.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The new apt-utils package appears to have been deployed on pepo (ftpmaster), so we can call this done from Launchpad's point of view.

apt (0.7.25.3ubuntu9.15~0.IS.10.04) lucid-cat; urgency=low

  [ Michael Vogt ]
  * Backport patch for apt-ftparchive to generate missing deb-src
    hashes. Unlike patch for Raring/Precise, only SHA1 and SHA256
    hashes are generated. (LP: #1078697)

 -- Barry Warsaw <email address hidden> Fri, 12 Apr 2013 11:36:54 -0400

We should consider this for lucid-updates too, so that we don't either lose this on security updates, or have to keep merging it.

Agreed, thanks for creating the bug task for Lucid.

I installed apt-utils version 0.8.16~exp12ubuntu10.11 from precise-proposed and confirm that the checksums are generated with it.

(precise-amd64)root@impulse:/home/bdmurray/test# apt-ftparchive sources .
  beef has no source override entry
  beef has no binary override entry either
Package: beef
Binary: beef
Version: 0.0.6-2
Maintainer: Andrea Bolognani <email address hidden>
Build-Depends: debhelper (>= 5.0.0)
Architecture: any
Standards-Version: 3.7.3
Format: 1.0
Files:
 4c6c354df2b5225d0c37790ed2b423bb 585 beef_0.0.6-2.dsc
 d991b00fba0067a7219cc926f777b8d3 14092 beef_0.0.6.orig.tar.gz
 6681e264aee438aba3f8723da6de9283 2031 beef_0.0.6-2.diff.gz
Homepage: http://www.kiyuko.org/beef
Checksums-Sha1:
 f6283f2f4393c360b52a7e905b79d493872635da 585 beef_0.0.6-2.dsc
 df43c00f538ee7d072c6b2b863557a9e390d0853 14092 beef_0.0.6.orig.tar.gz
 eca712929f4414ea0afe3849d4d9044401d49d9f 2031 beef_0.0.6-2.diff.gz
Checksums-Sha256:
 3c0d719aacd29eb1f0123295eb27a5dff780e9557c05ec8045fca6c75dedb2a5 585 beef_0.0.6-2.dsc
 ed23a998e3a50b0e4b3382b6415c3042c2cadd40b24c1a198bc309f9b3819217 14092 beef_0.0.6.orig.tar.gz
 01216e0972a4ede696388924e761f5ae909d46c06a038637ddf447bc64a3e2e8 2031 beef_0.0.6-2.diff.gz
Checksums-Sha512:
 8bda5c00d6e3319ad8a38a2d1af38bf2d3b49401efba36b5d7ce55da85e589faa3572aba52bd6d2eda01fe8a7b3f436ea9ecc97df08ac137ed9664c35ec0a685 585 beef_0.0.6-2.dsc
 fc2c8623b54c21415fa49b304d9de3d42b3de9b715d4d330af684921003d0512ac3c6ebee9f260336e99ddd5416c45b224a2ef6b82f5762d2d701d7a929c2bee 14092 beef_0.0.6.orig.tar.gz
 f51a016b8b800df6dab209cb17df3aba38a94a8792997e0d71b72c6804bc9298606d62a2d141d33ef7217ccd47721fb8f549be47cd6644fbd2911651069d8ee1 2031 beef_0.0.6-2.diff.gz

This bug was fixed in the package apt - 0.8.16~exp12ubuntu10.11

---------------
apt (0.8.16~exp12ubuntu10.11) precise-proposed; urgency=low

  [ Michael Vogt ]
  * test/integration/test-bug-1078697-missing-source-hashes:
    - add test for deb-src hash generation

  [ Marc Deslauriers ]
  * make apt-ftparchive generate missing deb-src hashes (LP: #1078697)
 -- Michael Vogt <email address hidden> Thu, 04 Apr 2013 18:42:34 +0200

Verification is done for precise so removing -needed tag.

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".