requires uncompressed Packages files on CDs

Bug #255545 reported by Colin Watson
28
This bug affects 2 people
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Fix Released
Medium
Michael Vogt
Hardy
Fix Released
Undecided
Michael Vogt

Bug Description

Binary package hint: apt

TEST CASE:
1. insert a intrepid alternate CD
2. open a temrinal and type "apt-cdrom add"
3. watch it fail

4. install the new apt from hardy-proposed
5. insert a intrepid alternate CD
6. type apt-cdrom add and verify that it works (its ok that it prints warnings about missing files)

----

When acquiring packages from a networked archive, apt is quite happy for the uncompressed Packages file to be omitted to save space; it only fetches the indexes it needs, and will try Packages.bz2 (if bzip2 is available) or Packages.gz first, so omitting Packages is harmless.

However, the same does not hold for CDs, which is rather unfortunate as the space problem is more acute there; we'd like to be able to remove the uncompressed Packages files from our alternate install CDs and save about 1.5MB. apt-cdrom wants to copy all the indexes, and gets upset when some of them are missing. Here's a demonstration with a hacked-up loop-mounted image from which I removed all the uncompressed Packages files:

  $ sudo apt-cdrom -m add
  Using CD-ROM mount point /cdrom/
  Identifying.. [661daa73dbd974b9ad9c1ffeeab8f8f0-2]
  Scanning disc for index files..
  Found 2 package indexes, 0 source indexes, 0 translation indexes and 1 signatures
  Found label 'Ubuntu 8.10 _Intrepid Ibex_ - Alpha i386 (20080806.2)'
  This disc is called:
  'Ubuntu 8.10 _Intrepid Ibex_ - Alpha i386 (20080806.2)'
  Copying package lists...gpgv: Signature made Wed 06 Aug 2008 16:39:54 BST using DSA key ID FBB75451
  gpgv: Good signature from "Ubuntu CD Image Automatic Signing Key <email address hidden>"
  E: Could not open file /cdrom/dists/intrepid/main/binary-i386/Packages - open (2 No such file or directory)
  E: Unable to determine the file size - fstat (9 Bad file descriptor)
  W: Hash mismatch for: main/binary-i386/Packages
  E: Could not open file /cdrom/dists/intrepid/main/debian-installer/binary-i386/Packages - open (2 No such file or directory)
  E: Unable to determine the file size - fstat (9 Bad file descriptor)
  W: Hash mismatch for: main/debian-installer/binary-i386/Packages
  E: Could not open file /cdrom/dists/intrepid/restricted/binary-i386/Packages - open (2 No such file or directory)
  E: Unable to determine the file size - fstat (9 Bad file descriptor)
  W: Hash mismatch for: restricted/binary-i386/Packages
  E: Could not open file /cdrom/dists/intrepid/restricted/debian-installer/binary-i386/Packages - open (2 No such file or directory)
  E: Unable to determine the file size - fstat (9 Bad file descriptor)

I suspect that it might be possible to work around this by removing the uncompressed Packages files from Release, but this would probably cause other problems. Last I checked, the size and checksums of Packages had to be in Release even if the files themselves were missing, in order that apt could verify those files after uncompressing them. People do create local mirrors starting from an Ubuntu CD image quite often, so I wouldn't like to introduce a two-way incompatibility here.

Would it be possible to only fail if none of the versions of a given Packages file are available, or explicitly copy only the compressed files, or synthesise the uncompressed files from the compressed ones in order to verify Release file integrity, or something?

Revision history for this message
Adam Conrad (adconrad) wrote :

I may be slightly naive here (having not looked into the code yet), but can we not have apt-cdrom reuse the apt-get codepath to "unpack Packages.{bz2,gz} to /var/lib/apt" and then use the results for the Release file verification?

One could, of course, make the above conditional on the CD not having uncompressed Packages files, if you want to avoid populating /var/lib/apt and wasting local disk space when the CD *does* have those files.

Revision history for this message
Michael Vogt (mvo) wrote :

Thanks for your bugreport.

I fixed this in my bzr branch by ignoring non-existing files on the CD. It will do the right thing when it comes to writing the packages file to the lists/ dir (uncompress it etc).

Changed in apt:
assignee: nobody → mvo
importance: Undecided → Medium
milestone: none → intrepid-alpha-4
status: New → Fix Committed
Revision history for this message
Martin Pitt (pitti) wrote :

Too late now, please upload right after Alpha-4.

Changed in apt:
milestone: intrepid-alpha-4 → intrepid-alpha-5
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 0.7.14ubuntu6

---------------
apt (0.7.14ubuntu6) intrepid; urgency=low

  * debian/apt.conf.autoremove:
    - remove "linux-image" (and friends) from the auto-remove
      blacklist. we have the kernel fallback infrastructure now
      in intrepid (thanks to BenC)
  * apt-pkg/indexcopy.cc:
    - support having CDs with no Packages file (just a Packages.gz)
      by not forcing a verification on non-existing files
     (LP: #255545)
  * apt-pkg/deb/dpkgpm.cc:
    - improve the filtering for duplicated apport reports (thanks
      to seb128 for pointing that problem out)
    - do not report disk full errors from dpkg via apport

 -- Michael Vogt <email address hidden> Thu, 07 Aug 2008 16:28:05 +0200

Changed in apt:
status: Fix Committed → Fix Released
Revision history for this message
Sarah Kowalik (hobbsee-deactivatedaccount) wrote :

Are we going to backport some form of this apt, so that upgrades from hardy, with an intrepid alternate CD, will work?

Revision history for this message
Colin Watson (cjwatson) wrote :

Sarah: that was filed separately as bug 267239, which I targeted to hardy a while back.

Revision history for this message
Sarah Kowalik (hobbsee-deactivatedaccount) wrote : Re: [Bug 255545] Re: requires uncompressed Packages files on CDs

Oh, my error.

Sorry for the noise :)

Michael Vogt (mvo)
Changed in apt:
assignee: nobody → mvo
status: New → In Progress
Michael Vogt (mvo)
description: updated
Revision history for this message
Martin Pitt (pitti) wrote :

I marked bug 267239 as a duplicate and will use this bug for SRU tracking, since Michael's upload refers to this one.

Changed in apt:
status: In Progress → Fix Committed
Revision history for this message
Martin Pitt (pitti) wrote :

Accepted into hardy-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Revision history for this message
Steve Beattie (sbeattie) wrote :

I can verify that apt 0.7.9ubuntu17 in 8.04 is unable to add an intrepid iso image and that apt 0.7.9ubuntu17.1 in hardy-proposed is able to do so. Further tests of apt from hardy-proposed showed that it was able to do additional installs, updates, and dependency calculation, but more verification for regressions would be appreciated.

Revision history for this message
Martin Pitt (pitti) wrote :

I upgraded to this apt version and did a range of standard operations, which all worked fine.

Since Intrepid will be released tomorrow, and this affects CD upgrades, I waive the 7 day period and copy it to -updates now.

Changed in apt:
status: Fix Committed → Fix Released
Revision history for this message
tz (thomas-mich) wrote :

Back in the Duplicate bug report (https://bugs.launchpad.net/bugs/267239), I asked if the dist-upgrade process or something else would insure an updated apt-cdrom before the intrepid upgrade or some other mechanism would insure hardy had this upgraded package BEFORE the intrepid upgrade (consider computers not on the internet). See the last comment in that other bug report.

I know if I was doing things, I wouldn't bother upgrading Hardy this soon before Intrepid is out, I'd just install Intrepid.

At minimum, some note on the CD-ROM image page where they are downloaded should say something about this if a non-upgraded hardy can't use a new intrepid install CD/DVD.

Revision history for this message
tz (thomas-mich) wrote :

Or to put it briefly, has anyone tried a NON-updated Hardy with a current Intrepid CD - especially one without an internet connection, but even one with access.

Revision history for this message
Eugenia Loli-Queru (eloli) wrote :

This bug is STILL THERE on Interpid. And in my opinion, it's a bloody big bug. I am dumbfounded that Canonical did not test this before the release.

So I burned an ISO of Ubuntu 8.10. I put it in to the drive, but it insisted on installing clean instead of offering me an upgrade to my existing Ubuntu installation on this laptop (which has the latest 8.04 version and updates installed). There is no Windows or OSX CD that wouldn’t offer me an upgrade.

Add the CD as a source package to my existing installation you say? Sure. This is what I get in this case:

    E:Could not open file /cdrom/dists/intrepid/main/binary-i386/Packages - open (2 No such file or directory), E:Unable to determine the file size - fstat (9 Bad file descriptor), W:Hash mismatch for: main/binary-i386/Packages, E:Could not open file /cdrom/dists/intrepid/restricted/binary-i386/Packages - open (2 No such file or directory), E:Unable to determine the file size - fstat (9 Bad file descriptor), W:Hash mismatch for: restricted/binary-i386/Packages

In other words, I have no upgrade path through the CD. I just spent 2 hours downloading the ISO, and 700 MBs from my Comcast monthly allowance. If you haven't heard, we can't use as much bandwidth as we want now with Comcast, we got to be careful. So I don't feel very good having downloaded the 700 MB ISO for nothing.

Revision history for this message
Eugenia Loli-Queru (eloli) wrote :

After rebooting the laptop, the error doesn't happen anymore (i guess apt needed a reload to get the updated code), but i still can't use the normal installation CD to upgrade. I need the alternate CD it seems. This sucks. It is completely illogical to not have an upgrade facility on the normal installation CDs.

Revision history for this message
David Marsh (rdmarsh) wrote :

Does anyone have a fix for this? I'm trying to upgrade server 8.04 to 8.10 and am seeing this problem when I "apt-cdrom add".

This server never connects to the internet (corporate lan), and all the upgrades have been through ISO images. I've done this successfully for the previous 2 releases or so.

Revision history for this message
tz (thomas-mich) wrote :

I haven't tried this specifically (as a work-around), but if you burn a recent Hardy daily build (dvd to get more packages?), it will have the apt-cdrom tool updated, and the installer should recognize it as having updates and do the upgrade of the packages. Then Intrepid should install without any problems.

The other "fix" is to decompress the Package files (extract/reburn the intrepid iso).

A third possible fix is to first upgrade the apt packages from the intrepid disk manually (dpkg -i /media/cdrom/...)

I don't think if you boot live it can upgrade a Hardy to Intrepid.

Revision history for this message
David Marsh (rdmarsh) wrote :

Thanks for your help tz.

Just updating back here on how I fixed this for others in the same boat:

I downloaded the apt deb package from here: http://packages.ubuntu.com/hardy-updates/apt

Then I copied it across to the server that didn't have internet connectivity and can only be updated using the cdrom images.

Then I ran "dpkg -i apt_0.7.9ubuntu17.1_i386.deb" as root.

Then I could "apt-cdrom add" successfully. I did get some errors about "Skipping non-exisiting file", but I could apt-get upgrade / apt-get dist-upgrade successfully to 8.10.

Revision history for this message
Colin Watson (cjwatson) wrote :

There'll be CD images of Ubuntu 8.04.2 fairly soon (this week, we hope) that you can use to upgrade without Internet access.

Revision history for this message
mathew (meta23) wrote :

This seems to be broken (again? still?) upgrading to Kubuntu 10.4 Lucid Lynx using apt-get 0.7.23.1ubuntu2.

# apt-cdrom add
Using CD-ROM mount point /cdrom/
Unmounting CD-ROM
Waiting for disc...
Please insert a Disc in the drive and press enter
Mounting CD-ROM...
Identifying.. [eca73cc0deabfb395b168f19b0f22193-2]
Scanning disc for index files..
Found 2 package indexes, 0 source indexes, 0 translation indexes and 1 signatures
This disc is called:
'Kubuntu 10.04 LTS _Lucid Lynx_ - Release i386 (20100427)'
Copying package lists...gpgv: Signature made Tue 27 Apr 2010 06:49:50 AM CDT using DSA key ID FBB75451
gpgv: Good signature from "Ubuntu CD Image Automatic Signing Key <email address hidden>"
Reading Package Indexes... Done
Writing new source list
Source list entries for this disc are:
deb cdrom:[Kubuntu 10.04 LTS _Lucid Lynx_ - Release i386 (20100427)]/ lucid main restricted
Unmounting CD-ROM...
Repeat this process for the rest of the CDs in your set.
W: Skipping non-exisiting file /cdrom/dists/lucid/main/binary-i386/Packages
W: Skipping non-exisiting file /cdrom/dists/lucid/restricted/binary-i386/Packages

apt-get update / dist-upgrade then fails to see anything on the CD.

Revision history for this message
Steve Beattie (sbeattie) wrote :

mathew,

Are you sure your download of the Kubuntu image is good?

$ sudo apt-cdrom add
Using CD-ROM mount point /cdrom/
Unmounting CD-ROM
Waiting for disc...
Please insert a Disc in the drive and press enter
Mounting CD-ROM...
Identifying.. [447cb85e73e74ab7dc77d29271638b9b-2]
Scanning disc for index files..
Found 2 package indexes, 0 source indexes, 0 translation indexes and 1 signatures
This disc is called:
'Kubuntu 10.04 LTS _Lucid Lynx_ - Release i386 (20100427)'
Copying package lists...gpgv: Signature made Tue 27 Apr 2010 03:14:02 AM PDT using DSA key ID FBB75451
gpgv: Good signature from "Ubuntu CD Image Automatic Signing Key <email address hidden>"
Reading Package Indexes... Done
Writing new source list
Source list entries for this disc are:
deb cdrom:[Kubuntu 10.04 LTS _Lucid Lynx_ - Release i386 (20100427)]/ lucid main restricted
Unmounting CD-ROM...
Repeat this process for the rest of the CDs in your set.
W: Skipping non-exisiting file /cdrom/dists/lucid/main/binary-i386/Packages
W: Skipping non-exisiting file /cdrom/dists/lucid/main/debian-installer/binary-i386/Packages
W: Skipping non-exisiting file /cdrom/dists/lucid/restricted/binary-i386/Packages
W: Skipping non-exisiting file /cdrom/dists/lucid/restricted/debian-installer/binary-i386/Packages
$ grep "^deb cdrom" /etc/apt/sources.list
deb cdrom:[Kubuntu 10.04 LTS _Lucid Lynx_ - Release i386 (20100427)]/ lucid main restricted

$ sudo apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
[SNIP]
719 upgraded, 65 newly installed, 147 to remove and 1 not upgraded.
Need to get 0B/387MB of archives.
After this operation, 240MB disk space will be freed.
Do you want to continue [Y/n]?

$ apt-cache policy apt
apt:
  Installed: 0.7.23.1ubuntu2
  Candidate: 0.7.25.3ubuntu7
  Version table:
     0.7.25.3ubuntu7 0
        500 cdrom://Kubuntu 10.04 LTS _Lucid Lynx_ - Release i386 (20100427) lucid/main Packages
 *** 0.7.23.1ubuntu2 0
        500 http://denisovich karmic/main Packages
        100 /var/lib/dpkg/status

Admittedly, this is with a karmic ubuntu desktop install, but which
desktop environment it is installed under should not matter to apt.

--
Steve Beattie
<email address hidden>
http://NxNW.org/~steve/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.