[dapper->hardy] update-manager for edgy needs to upgrade dpkg/apt before calculating the upgrade to support the new "breaks" - otherwise the upgrade may fail

Thanks for this reminder.

I was looking into this today and while this is certainly possible. But I'm concerned that the addition of breaks may break the dist-upgrader in much the same way as what happend to firefox and the firefox-theme package.

The problem is essentially the same, i.e. the libapt implementation that calculates the upgrade from dapper->edgy will not know about breaks (yet). Downloading/install dpkg early does not help here because libapt needs some understanding about the field as well. Upgrading dpkg/apt and then re-runing the dist-upgrader to calculate a new upgrade would work but it makes the operation irreversible (well, we could DOWNGRADE again *shudder*) - something that we don't really want.

I see the following possible strategies:
1. put a updated apt into dapper-updates that supports breaks (and if we do this anyway add dpkg as well to avoid the extra step of the dist-upgrader). This shouldn't be too bad because presumably this changes does not change anything for packages without breaks-fields. OTOH it is going to be a invasive change in dapper (something we may not want to do after the X fiasco)
2. put a updated apt into dapper-updates that maps "breaks" to "conflicts". This means we don't win anything but we don't lose anything either. The only advantage here is that the change in apt is likely to be minimal (~5 lines of code hopefully)
3. bite the bullet and start using breaks in edgy :(

None of those is very applealing. I'm open for suggestions and better ideas of course :)

Cheers,
 Michael

Michael Vogt writes ("[Bug 54234] Re: update-manager for edgy needs to upgrade dpkg early on"):
> The problem is essentially the same, i.e. the libapt implementation that
> calculates the upgrade from dapper->edgy will not know about breaks
> (yet). Downloading/install dpkg early does not help here because libapt
> needs some understanding about the field as well. Upgrading dpkg/apt and
> then re-runing the dist-upgrader to calculate a new upgrade would work
> but it makes the operation irreversible (well, we could DOWNGRADE again
> *shudder*) - something that we don't really want.

So as I understand it, the problem is that the dist-upgrader wants to
calculate in advance what it's going to do, before giving the user the
chance to say yes/no ? And you can't calculate correctly with the old
apt.

How about unpacking the new apt somewhere and running it with
LD_PRELOAD et al to calculate the upgrade ?

Ian.

Just a quick update on this bug.

The problem is indeed that the latest apt/python-apt is required to correctly calculate the upgrade in advance. We need to do it in advance to make sure that it will work and to give the user the chance to cancel it if he does not like what he sess (e.g. the amount of data to download).

After trying various methods we think that the only solution to support this is via apt/python-apt from dapper-backports. The tool would have to download those, install them into a tempdir and use LD_PRELOAD/LD_LIBRARY_PATH/PYTHONPATH to ensure that they are used. Currently this is blocked on a bug in soyuz that prevents backports from working.

The alternative is of course to not use breaks until edgy+1 (which would be unfortunate, but given the current situation with the lack of a backported apt and the lack of time until release it may well be the only feasible way).

Cheers,
 Michael

Michael Vogt writes ("[Bug 54234] Re: update-manager for edgy needs to upgrade dpkg/apt before calculating the upgrade to support the new "breaks""):
> The alternative is of course to not use breaks until edgy+1 (which would
> be unfortunate, but given the current situation with the lack of a
> backported apt and the lack of time until release it may well be the
> only feasible way).

There have been at least two cases of Breaks being used in the archive
that I know of, one of which would be tedious to back out.

I think the real alternative is a special non-auto-built version of
the libapt and python-apt packages.

Ian.

While looking at the issue again today I noticed another thing that complicates the matters. The new apt in backports would have a different ABI than the one in dapper. So a backport would mean we need rebuilds or backports of all rdependencies of apt as well.

Michael Vogt writes ("[Bug 54234] Re: update-manager for edgy needs to upgrade dpkg/apt before calculating the upgrade to support the new "breaks""):
> While looking at the issue again today I noticed another thing that
> complicates the matters. The new apt in backports would have a different
> ABI than the one in dapper. So a backport would mean we need rebuilds or
> backports of all rdependencies of apt as well.

The only backports we need for the upgrade-manager are libapt and
python-apt, right ?

Ian.

This is correct. The trouble is that if we move it into dapper-backports it will be used by a lot of people interessted in backports and it will not only be availalbe for the upgrader.

These people will probably not like a half-backported stack but rather want to have the complete one (including synaptic, adept). OTOH the list is not that big and if synaptic + adept are backported that should make most people happy.

It looks like for the beta we need to got with the "build by hand" and "host on people.u.c" solution :)

Next problem. For CDROM upgrades (that should be done without network) we won't be able to get a backported apt from the net.

Solutions:
a) don't have packages with "Breaks" on the CD
b) put a backported apt/python-apt onto the CD

This is precisely the reason why we ship the upgrader as a tarball rather than as a package.

If it requires additional tools which aren't available in the old version, they should be shipped with the upgrader. This would seem to address all of the issues described here.

Note that shipping an updated apt and dpkg in -updates has exactly the same issues as doing a partial upgrade first, in that it is irreversible. Furthermore, this upgrade would be applied for users even if they never upgraded, in which case it is unnecessary.

So, following discussion on IRC, as I understand it the things that
still need to be done are:

  * Put apt/dpkg/python-apt into dapper-backports; AIUI we have the
    appropriate sources for these packages ready and we have (or can
    easily have) binary builds for testing - for now, the
    update-manager can download these from people.u.c or similar, and
    this will be rectified when soyuz backports are fixed.

  * Arrange for update-manager to look on the CD for these files, as
    well as being able to download them (the latter is already done).

  * Arrange for the CD build to contain these files in the appropriate
    place on the CD as required by update-manager.

Ian.

Here is the current status for this bug:
- a new version of the dist-upgrader is uploaded. it supports fetching of selected backports from arbitrary sources.list fragments (e.g. dapper-backports)
- it will get and unpack those packages into a tempdir and modify PATH,LD_LIBRARY_PATH and PYTHONPATH to make sure that the updated versions are used
- it will re-exec itself once the update is fetched to support the new libraries
- it will still work with --cdrom, even if no updates are found on the cdrom if a network is used

Still todo:
1. get the backports into the official dapper-backports
2. add the backports to the CD
3. make dist-upgrader look for the backports on CD first (to support upgrades without network)

@mdz:
If we would ship the updated tools in the tarball that would indeed solve the problems.

To do this, we would still need to build a backported libapt/dpkg on a dapper system (because of new glib2.4 symbols it won't run on dapper when build on edgy) and put that in the tarball.

The tarball itself would need to be placed in edgy/dist-upgrader-$(ARCH). Currently soyuz custom upload-format code for this is not able to deal with a arch-dependant dist-upgrader unfortunately. The update-manager assumes a ARCH=all as well (but this would be trivial to fix and put into dapper-updates).

My current solution for this is outlined above. To me it looks not optimal either way. Putting it all into a tarball make the code easier but it involves more work with the build-system and additional soyuz work.

I would appreciate your input on this (which of the two poisons to pick).

Cheers,
 Michael

On Fri, Sep 22, 2006 at 11:12:04PM -0000, Michael Vogt wrote:
> Here is the current status for this bug:
> - a new version of the dist-upgrader is uploaded. it supports fetching of selected backports from arbitrary sources.list fragments (e.g. dapper-backports)
> - it will get and unpack those packages into a tempdir and modify PATH,LD_LIBRARY_PATH and PYTHONPATH to make sure that the updated versions are used
> - it will re-exec itself once the update is fetched to support the new libraries
> - it will still work with --cdrom, even if no updates are found on the cdrom if a network is used
>
> Still todo:
> 1. get the backports into the official dapper-backports
> 2. add the backports to the CD
> 3. make dist-upgrader look for the backports on CD first (to support upgrades without network)

I don't like the idea of relying on dapper-backports for this; I think we
should ship the upgrader complete and independent of any external packages.

--
 - mdz

Here is a list of the steps that needs to be taken to make sure that a "arch != all" dist-upgrader works. The good thing about this is that it can be done in parallel to the current (working) implementation that fetches the backports from the net.

Steps:
- add new "UpgradeToolBaseURI" key to the meta-release file at http://changelogs.ubuntu.com/meta-release that contains the base-path to the upgrader
- change DistUpgradeFetcher.py in update-manager--dapper to honor the new UpgradeToolBaseURI from the meta-release file and add "binary-$(dpkg --print-architecture)/current" to it
- patch soyuz to accept raw-dist-upgrade uploads that are not arch==all and move them to "edgy/dist-upgrader/binary-$arch/$version" (+ keeping a current symlink)
- change the dist-upgrader to check/use the backports in the tarbal
- change the build-system of the dist-upgrader to make sure to include the right backported binary bits
- change the update-notifier dist-upgrader on cdrom detection to search for CDROM_ROOT/dists/stable/dist-upgrader/binary-$(arch)
- change the cd-build script (tools/edgy/upgrade.sh) to place the tarball into the CDROM_ROOT/dists/stable/dist-upgrader/binary-$(arch) subdir

We can do all that in parallel to the current solution and only switch with the final update-manager backport upload. Does that sound ok?

Cheers,
 Michael

On Tue, Sep 26, 2006 at 02:18:24PM -0000, Michael Vogt wrote:
> Here is a list of the steps that needs to be taken to make sure that a
> "arch != all" dist-upgrader works. The good thing about this is that it
> can be done in parallel to the current (working) implementation that
> fetches the backports from the net.
>
> Steps:
> - add new "UpgradeToolBaseURI" key to the meta-release file at http://changelogs.ubuntu.com/meta-release that contains the base-path to the upgrader
> - change DistUpgradeFetcher.py in update-manager--dapper to honor the new UpgradeToolBaseURI from the meta-release file and add "binary-$(dpkg --print-architecture)/current" to it
> - patch soyuz to accept raw-dist-upgrade uploads that are not arch==all and move them to "edgy/dist-upgrader/binary-$arch/$version" (+ keeping a current symlink)
> - change the dist-upgrader to check/use the backports in the tarbal
> - change the build-system of the dist-upgrader to make sure to include the right backported binary bits
> - change the update-notifier dist-upgrader on cdrom detection to search for CDROM_ROOT/dists/stable/dist-upgrader/binary-$(arch)
> - change the cd-build script (tools/edgy/upgrade.sh) to place the tarball into the CDROM_ROOT/dists/stable/dist-upgrader/binary-$(arch) subdir
>
> We can do all that in parallel to the current solution and only switch
> with the final update-manager backport upload. Does that sound ok?

Hmm, I didn't realize we didn't have support for arch-any upgrader tarballs
yet.

In light of this, and the beta release, I think we should take a
conservative approach and not use Breaks in main. It seems to be used in
only 3 packages at present, and these can be reverted to conflicts for now.

Both the fetching of backports from the network, and support for backports
in the tarball are useful features that we want, but we'll have to put them
on hold for Edgy.

Please ensure that we have upgrade instructions ready for inclusion in the
beta announcement, as this is our best chance for pre-release upgrade
testing.

--
 - mdz

Removed the beta-blocker tag. We do not ship any package that uses Breaks so this is not required for the dapper->edgy upgrade.

hi
will trying to upgrade (gksu "upgrade-manager -c -d") from dapper to edgy I got this error:

"could not calculate the upgrade"

I am attaching the bugreport

can you help? I just want to try the new release

Solution : Remove the "libgl1-mesa-dri" package from Beerorkid repository "deb http://www.beerorkid.com/compiz/ dapper main" which breaks the update.

gksu "upgrade-manager -c -d" should then work.

thanks
I'll try removing that.

On Fri, Sep 29, 2006 at 10:02:52AM -0000, Andreas Constantinides wrote:
> will trying to upgrade (gksu "upgrade-manager -c -d") from dapper to edgy I got this error:

This is unrelated to this bug; please avoid discussing other problems in
this bug report as it causes confusion for other users who may read it.

--
 - mdz

The current version of update-manager does support this for dapper->hardy upgrades.