pycompile fixes needed for maverick

Bug #689615 reported by Matthias Klose
36
This bug affects 7 people
Affects Status Importance Assigned to Milestone
python-defaults (Ubuntu)
Fix Released
Undecided
Unassigned
Maverick
Fix Released
High
Unassigned
python2.7 (Ubuntu)
Fix Released
High
Unassigned
Maverick
Invalid
Undecided
Unassigned
update-manager (Ubuntu)
Fix Released
High
Unassigned
Maverick
Invalid
Undecided
Unassigned

Bug Description

some pycompile backports are needed to allow smooth maverick -> natty upgrades. This changes are conservative, and the same that are already in Debian squeeze or considered for it.

some of the failing upgrade reports are: 687658, 688846, 688965, 689103, 689306

the changes are limited to pycompile, only used by a few packages built by dh_python2

Revision history for this message
Martin Pitt (pitti) wrote :

Is there any chance we could at least mitigate this in the natty preinst as well? We can't really guarantee that everyone installs maverick-updates before upgrading to natty.

Also, is this a bug in natty itself? If not, please close the natty task.

Revision history for this message
Matthias Klose (doko) wrote :

fixed in natty.

I don't see a way to fix this in the preinst. pycompile get's called from python2.7/python2.7-minimal while the fixed python/python-minimal is not yet installed.

Changed in python-defaults (Ubuntu):
status: New → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote :

> pycompile get's called from python2.7/python2.7-minimal

I don't understand this -- you mean when you install a python 2.7 enabled package, it calls pycompile (which is shippped by "python-minimal", not "python2.7" or "python2.7-minimal), while python-minimal is still at 2.6? If that's the case, why can't python2.7 just add a "Breaks: python-minimal (<< 2.7)"?

Revision history for this message
Matthias Klose (doko) wrote : Re: [Bug 689615] Re: pycompile fixes needed for maverick -> natty upgrades

On 13.12.2010 16:07, Martin Pitt wrote:
>> pycompile get's called from python2.7/python2.7-minimal
>
> I don't understand this -- you mean when you install a python 2.7
> enabled package, it calls pycompile (which is shippped by "python-
> minimal", not "python2.7" or "python2.7-minimal), while python-minimal
> is still at 2.6? If that's the case, why can't python2.7 just add a
> "Breaks: python-minimal (<< 2.7)"?

what would a "Breaks" change?

Revision history for this message
Martin Pitt (pitti) wrote : Re: pycompile fixes needed for maverick -> natty upgrades

The uploaded diff is rather large, and I'm trying to understand it. The referenced bugs all fail on "Requested versions are not installed". So that's a check from pycompile and a package requests 2.7, but since we still have the old python-minimal installed it doesn't know about this version yet?

The changes to public_modules.rtinstall and public_modules.rt remove seem irrelevant for this SRU, as we don't support 2.4/2.5 in maverick, right?

To me it seems the gut of the required change here is that, is this correct?

@@ -235,7 +255,13 @@
             'only, please use /usr/share/python/bcep to specify '
             'public modules to skip')

- versions = get_requested_versions(options.vrange, available=True)
+ if options.vrange and options.vrange[0] == options.vrange[1] and\
+ options.vrange != (None, None) and\
+ exists("/usr/bin/python%d.%d" % options.vrange[0]):
+ # specific version requested, use it even if it's not in SUPPORTED
+ versions = set(options.vrange[:1])
+ else:
+ versions = get_requested_versions(options.vrange, available=True)
     if not versions:
         log.error('Requested versions are not installed')
         exit(3)

What other part does actually specify the vrange option?

Is the addition of the --force and -O options related in any way?

Changed in python-defaults (Ubuntu Maverick):
status: New → Incomplete
Revision history for this message
Martin Pitt (pitti) wrote :

> what would a "Breaks" change?

You couldn't upgrade into a situation where you have python2.7-minimal installed with python-minimal version 2.6. My question was whether this was the very thing which would cause that bug here, as we can fix that with proper dependencies (and for everyone upgrading to Natty, not just for those who happen to install all maverick-updates first).

Revision history for this message
Matthias Klose (doko) wrote :

On 13.12.2010 16:21, Martin Pitt wrote:
> The uploaded diff is rather large, and I'm trying to understand it.

backporting fixes.

> The
> referenced bugs all fail on "Requested versions are not installed". So
> that's a check from pycompile and a package requests 2.7, but since we
> still have the old python-minimal installed it doesn't know about this
> version yet?
>
> The changes to public_modules.rtinstall and public_modules.rt remove
> seem irrelevant for this SRU, as we don't support 2.4/2.5 in maverick,
> right?
>
> To me it seems the gut of the required change here is that, is this
> correct?
>
> @@ -235,7 +255,13 @@
> 'only, please use /usr/share/python/bcep to specify '
> 'public modules to skip')
>
> - versions = get_requested_versions(options.vrange, available=True)
> + if options.vrange and options.vrange[0] == options.vrange[1] and\
> + options.vrange != (None, None) and\
> + exists("/usr/bin/python%d.%d" % options.vrange[0]):
> + # specific version requested, use it even if it's not in SUPPORTED
> + versions = set(options.vrange[:1])
> + else:
> + versions = get_requested_versions(options.vrange, available=True)
> if not versions:
> log.error('Requested versions are not installed')
> exit(3)
>
> What other part does actually specify the vrange option?

did I say that?

the changelog is quiet explicit about the changes.

> Is the addition of the --force and -O options related in any way?

yes, pycompile currently silently doesn't do anything when a .pyc file exists, which is wrong. It does the wrong thing for .pyo files, and it doesn't allow forcing byte compilation. These options are what a user of the py_compile standard module would expect.

And it's certainly wrong to ignore the exit status of the compileall subprocesses.

Changed in python-defaults (Ubuntu Maverick):
status: Incomplete → New
Revision history for this message
Martin Pitt (pitti) wrote :

> did I say that?

You didn't; as you didn't explain what the solution _is_, I was trying to infer it from the diff.

> the changelog is quiet explicit about the changes.

The changelog doesn't attribute any particular change entry with this bug, and most changes apply to the pycompile program, and the changelog doesn't specify which part of it (function name or so), it's also hard to say. It seems to me that the entire second block in the changelog are extra changes which are unrelated to this bug.

This would go a lot faster if this update would actually follow the SRU procedure..

Revision history for this message
Martin Pitt (pitti) wrote :

I read the entire diff once again, and most changes seem plausible to me. But you seem to imply that the code section in point 5 was not the one responsible for fixing this bug. So which is it then?

Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Accepted python-defaults into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in python-defaults (Ubuntu Maverick):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
Martin Pitt (pitti) wrote : Re: pycompile fixes needed for maverick -> natty upgrades

Adding python2.7 task. Above python-defaults change is just a bandaid, as it won't help upgraders to natty who don't install maverick-updates first. A first attempt at this has been made in

python2.7 (2.7.1-1ubuntu2) natty; urgency=low

  * python2.7-minimal: Add break on python-minimal (<< 2.7.1~).
    LP: #689615.

 -- Matthias Klose <email address hidden> Mon, 13 Dec 2010 19:17:03 +0100

but that now causes both packags to be held back on upgrades. Investigation about this is ongoing.

What we really need to do is to prevent configuring python2.7{,.-minimal} while we still have python-minimal (<< 2.7) installed.

Revision history for this message
Matthias Klose (doko) wrote : Re: [Bug 689615] Re: pycompile fixes needed for maverick -> natty upgrades

On 14.12.2010 17:07, Martin Pitt wrote:
>> did I say that?
>
> You didn't; as you didn't explain what the solution _is_, I was trying
> to infer it from the diff.

citation out of context. there are three questions, not just the last one.

>> the changelog is quiet explicit about the changes.
>
> The changelog doesn't attribute any particular change entry with this
> bug, and most changes apply to the pycompile program, and the changelog
> doesn't specify which part of it (function name or so), it's also hard
> to say. It seems to me that the entire second block in the changelog are
> extra changes which are unrelated to this bug.

fine. I'll retitle the bug.

> This would go a lot faster if this update would actually follow the SRU
> procedure..

All changes are appropriate for a SRU, and were made in natty before filing this
issue. And I think it is not appropriate to insist on untested
dependency/breaks changes, which I did tell you were no solution, and delay the
fix for real issues, and which did cause new issues in natty.

 > The changes to public_modules.rtinstall and public_modules.rt remove
 > seem irrelevant for this SRU, as we don't support 2.4/2.5 in maverick,
 > right?

please re-read the patch and the changelog:

   * Don't touch the standard python library in rtupdate scripts.

even maverick has a site directory.

the change for the particular upgrade problem is

   * pycompile:
     - if installed Python is requested via -V option, use it even if
       it's not in a list of supported Python versions

Yes, having this fix in maverick doesn't solve the upgrade problem for
installations with disabled -updates, but it will avoid us tons of bug reports
for installations with enabled -updates.

summary: - pycompile fixes needed for maverick -> natty upgrades
+ pycompile fixes needed for maverick
Revision history for this message
Matthias Klose (doko) wrote :

verified that an upgrade from maverick-proposed to natty works
verified that timestamps are respected with byte compilation
verified that the error code is propagated

tags: added: verification-done
removed: verification-needed
Changed in python2.7 (Ubuntu Maverick):
status: New → Invalid
Martin Pitt (pitti)
Changed in python2.7 (Ubuntu):
importance: Undecided → High
Revision history for this message
Michael Vogt (mvo) wrote :

I will make the release upgrader patch pycompile when the upgrade starts, we also need to release note that.

Changed in update-manager (Ubuntu):
status: New → Confirmed
milestone: none → ubuntu-11.04-beta
Changed in update-manager (Ubuntu Maverick):
status: New → Invalid
Changed in update-manager (Ubuntu):
importance: Undecided → High
Revision history for this message
Matthias Klose (doko) wrote :

please consider promoting the update to -updates without the usual ten day delay

Changed in python-defaults (Ubuntu Maverick):
importance: Undecided → High
milestone: none → maverick-updates
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-defaults - 2.6.6-2ubuntu2

---------------
python-defaults (2.6.6-2ubuntu2) maverick-proposed; urgency=low

  * Backport pycompile fixes to allow upgrades to natty. LP: #689615.

  [ Piotr Ożarowski ]
  * pycompile:
    - if installed Python is requested via -V option, use it even if
      it's not in a list of supported Python versions
    - `pycompile $DESTDIR/usr/lib/python*` will recognize public
      site-packages directories and use the right interpreter instead of
      raising KeyError
    - do not try to check if interpreter is present when version range is
      empty (closes: 605356)

  [ Matthias Klose ]
  * pycompile:
    - Don't over-optimize, check the timestamps of byte-code files.
    - Add options -f/--force, -O.
    - Propagate the exit value of the py_compile processes.
    - Update manual page.
  * Bump depends to require the fixed pycompile for dh_python2 based package
    builds.
  * Don't touch the standard python library in rtupdate scripts.
 -- Matthias Klose <email address hidden> Mon, 13 Dec 2010 12:36:23 +0100

Changed in python-defaults (Ubuntu Maverick):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package update-manager - 1:0.145.10

---------------
update-manager (1:0.145.10) natty; urgency=low

  * DistUpgrade/DistUpgradeQuirks.py:
    - fixup internal (ed) based patching tool
  * DistUpgrade/patches/
    - add pycompile patch to ensure clean upgrade even when maverick-updates
      is not available (LP: #689615)
  * AutoUpgradeTester/profile/server/DistUpgrade.cfg:
    - updated for maverick
  * DistUpgrade/DistUpgradeCache.py:
    - minor python-apt 0.8 API update
  * tests/patchdir/_patchdir_foo.f41121a903eafadf258962abc57c8644:
    - update test for latest internal patching tool
 -- Michael Vogt <email address hidden> Fri, 17 Dec 2010 18:24:36 +0100

Changed in update-manager (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Michael Vogt (mvo) wrote :

Update-manager carries the required patch and will apply it automatically if needed.

Revision history for this message
Murray Cumming (murrayc) wrote :

> This bug was fixed in the package update-manager - 1:0.145.10

I don't see this version yet. Should I? I have 1:0.142.20.

Revision history for this message
Murray Cumming (murrayc) wrote :

Ah, I do see that version, but I can't find a way to install it before apt-get tries to install other stuff that hits this problem, so I never get as far as upgrading update-manager itself.

Revision history for this message
Michael (mstrelan) wrote :

Can someone please confirm, do you need to update all Maverick packages before upgrading to Natty? I have a fresh Maverick install and skipped all the updates and upgraded to Natty. I got the same error in the OP. My only change to anything from stock Ubuntu was that I changed the mirror to iiNet Australia.

Matthias Klose (doko)
Changed in python2.7 (Ubuntu):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.