linaro-image-tools (hwpack-create) broken on oneiric 64-bits

I'm having a look at this; it's something to do with the list of conflicting packages.

expected package has: conflicts="baz | zap"
observed package has: conflicts="baz | baz | zap, zap"

As expected, this is because of a change in python-apt. On 32-bits Oneiric, in apt.package.Version.get_dependencies(), self._cand.depends_list has the expected conflicts list:

  {'Conflicts': [[<apt_pkg.Dependency object: pkg:'baz' ver:'' comp:''>, <apt_pkg.Dependency object: pkg:'zap' ver:'' comp:''>]],...}

Whereas on 64-bits, it has the following:

  {'Conflicts': [[<apt_pkg.Dependency object: pkg:'baz' ver:'' comp:''>, <apt_pkg.Dependency object: pkg:'baz' ver:'' comp:''>, <apt_pkg.Dependency object: pkg:'zap' ver:'' comp:''>], [<apt_pkg.Dependency object: pkg:'zap' ver:'' comp:''>]],...}

And in both cases this content is taken from an identical Packages entry:

Package: top
Version: 1.0
Filename: top_1.0_all.deb
Size: 26
Architecture: all
Depends: foo
Pre-Depends: bar (>= 1.0)
Conflicts: baz | zap
Recommends: zing, zang
MD5sum: d58702017db565c1fb359e2026f71088

It's a bug in APT's handling of Multi-Arch (the multiple baz's are from different architectures; see implicit conflicts in the Spec) -- the more interesting question is through why the hell you want to specify an or-group in Conflicts…

Thankfully, in my interpretation of the debian-policy §7.1 it isn't even allowed to specify or-groups in Conflicts.

Thanks for the clarification, David!

Is the bug you mention the fact that we get multiple baz's in Conflicts even though they have architecture:all?

Good question about the or-group conflicts. This is an existing test which I didn't write, but I can't think of any reason why this would be necessary, so I can turn that into "baz, zap". Although right now that still causes the conflicts list to be different between i386 and amd64

One thing I didn't realize until now is that this failing test that I'm debugging should not be affected by multi-arch because it uses a separate cache configured for armel, which AIUI, should not have multi-arch. After realizing this and digging a bit more, I've came up with the test-case below, where apt_pkg.Cache.is_multi_arch is set to True even though we set the architecture to armel. This seems to be because the rootdir passed to apt.cache.Cache is not passed down to apt_pkg.Cache, so the latter is reading the configs from the system.

import os, tempfile
from apt.cache import Cache

tempdir = tempfile.mkdtemp(prefix="hwpack-apt-cache-")
os.makedirs(os.path.join(tempdir, "etc", "apt"))
apt_conf = os.path.join(tempdir, "etc", "apt", "apt.conf")
with open(apt_conf, 'w') as f:
    f.write(
        'Apt {\nArchitecture armel;\n'
        'Install-Recommends "true";\n}\n')

c = Cache(rootdir=tempdir, memonly=True)
print "Cache._have_multi_arch: ", c._have_multi_arch
print "Cache._cache.is_multi_arch: ", c._cache.is_multi_arch

Forgot to mention that on the snipped I pasted above, Cache._have_multi_arch is False on both i386 and amd64 (because the arch is set to armel manually), whereas Cache._cache.is_multi_arch is False on i386 and True on amd64

And I was wrong when I said this was probably because apt.cache.Cache doesn't pass the rootdir down to apt_pkg.Cache. What happens in fact is that apt.cache.Cache reads the config from the rootdir into apt_pkg.config and I'd expect that apt_pkg.Cache would see that the Architecture is armel and thus there's no multi-arch needed.

The bug here is that apt calls "dpkg" from the non-rootdir for the "--print-foreign-architectures" call.

A easy workaround is to set:
  apt.apt_pkg.config.set("dir::bin::dpkg", "/bin/false")
in your script until apt is properly fixed.

A possible fix would be to set dir::bin::dpkg in python-apt to the rootdir dpkg. This would break for the case if dpkg is used to install packages inside the rootdir without the use of apt::chroot-directory. Not sure if that will actually work anyway.

Status changed to 'Confirmed' because the bug affects multiple users.

This bug was fixed in the package python-apt - 0.8.3ubuntu1

---------------
python-apt (0.8.3ubuntu1) precise; urgency=low

  * merged from debian/sid

python-apt (0.8.3) unstable; urgency=low

  [ Alexey Feldgendler ]
  * handle architecture-specific conflicts correctly (LP: #829138)

  [ Michael Vogt ]
  * lp:~mvo/python-apt/debfile-multiarch:
    - add multiarch support to the debfile.py code (LP: #854839)
  * tests/test_apt_cache.py:
    - add additional check if provides test can actually be run

python-apt (0.8.2) unstable; urgency=low

  [ Michael Vogt ]
  * merged from ubuntu:
    - use logging instead of print
    - update distro template Ubuntu.info.in
    - add xz compression support
  * po/python-apt.pot:
    - refreshed
  * po/pt_BR.po:
    - updated, thanks to Sergio Cipolla (closes: #628398)
  * po/da.po:
    - updated, thanks to Joe Dalton (closes: #631309)
  * po/sr.po:
    - updated, thanks to Nikola Nenadic (closes: #638308)
  * python/apt_pkgmodule.cc:
    - add apt_pkg.get_architectures() call
  * apt/cache.py:
    - remove "print" when creating dirs in apt.Cache(rootdir=dir),
      thanks to Martin Pitt
    - add __lt__ to apt.Package so that sort() sorts by name
      on list of package objects
  * debian/control:
    - add recommends to xz-lzma to ensure we have the unlzma command
  * apt/cache.py:
    - set Dir::bin::dpkg if a alternate rootdir is given
      (LP: #885895)
  * build fixes for the apt in experimental
  * apt/debfile.py:
    - raise error when accessing require_changes and
      missing_deps without calling check() before, thanks to
      Tshepang Lekhonkhobe (closes: #624379)
  * apt/package.py:
    - add new "suggests" property, thanks to Christop Groth
    - allow Dependency object to be iteratable, this allows to write
      code like:
      for or_dep_group in pkg.candidate.dependencies:
          for dep in or_dep_group:
              do_something()
      (thanks to Christop Groth)
  * apt/progress/base.py:
    - write exception text to stderr to avoid hidding exceptions
      like "pre-configure failed" from libapt (thanks to Jean-Baptiste
      Lallement)

  [ Tshepang Lekhonkhobe ]
  * rm usage of camelcase in cache.py doc (closes: #626617)
  * grammar fix in the cache.py doc (closes: #626610)

  [ Nikola Pavlović ]
  * fixed a typo, changed "Open Source software" to
    "free and open-source software" (LP: #500940)
 -- Michael Vogt <email address hidden> Thu, 08 Dec 2011 20:44:30 +0100

Thanks a lot for fixing this, Michael! I've just confirmed that on
Precise we no longer need our hack in linaro-image-tools to workaround
this.

Now, should we expect this fix to be backported to Oneiric as well?