failing pypy jobs with "option --single-version-externally-managed not recognized"

Bug #1290562 reported by Jeremy Stanley
30
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Core Infrastructure
Fix Released
High
Jeremy Stanley
Solum
Won't Fix
Undecided
Unassigned
python-ceilometerclient
Fix Released
Undecided
Ildiko Vancsa
python-neutronclient
Fix Released
Medium
Akihiro Motoki

Bug Description

[...]
   Found existing installation: setuptools 2.2
     Uninstalling setuptools:
       Successfully uninstalled setuptools
   Running setup.py install for mccabe
     /usr/lib/pypy/lib-python/2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'
       warnings.warn(msg)
2014-03-10 08:19:39.567 | /usr/lib/pypy/lib-python/2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'zip_safe'
       warnings.warn(msg)
     usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
        or: -c --help [cmd1 cmd2 ...]
        or: -c --help-commands
        or: -c cmd --help

     error: option --single-version-externally-managed not recognized
     Complete output from command /home/jenkins/workspace/gate-python-novaclient-pypy/.tox/pypy/bin/pypy -c "import setuptools, tokenize;__file__='/home/jenkins/workspace/gate-python-novaclient-pypy/.tox/pypy/build/mccabe/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/tmp.l53ekBaPmF/pip-5i1oDp-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/jenkins/workspace/gate-python-novaclient-pypy/.tox/pypy/include/site/python2.7:
     /usr/lib/pypy/lib-python/2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'

   warnings.warn(msg)

 /usr/lib/pypy/lib-python/2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'zip_safe'

   warnings.warn(msg)

 usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]

    or: -c --help [cmd1 cmd2 ...]

    or: -c --help-commands

    or: -c cmd --help

 error: option --single-version-externally-managed not recognized
[...]

http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_567

Tags: gate-failure
Revision history for this message
Jeremy Stanley (fungi) wrote :

This seems to fail consistently on workers in rax but succeed in hpcloud. We saw it briefly weeks ago, but the release of setuptools 2.2 seemed to cause it to cease. The most recent occurrences seem to possibly coincide with Saturday's release of setuptools 3.1, but that could simply be a red herring. So far I've been unable to identify the particular difference discriminating this failure between these systems.

I have narrowed it down to a simple means of recreating on rax: within a pypy virtualenv, use pip to upgrade setuptools and mccabe together on the same command line. Examples...

Working in hpcloud: http://paste.openstack.org/show/73058/

Failing in rax: http://paste.openstack.org/show/73060/

I discussed this with Alex Gaynor just now, and we agree that this feels like some sort of order of execution strangeness in pip which is getting triggered by hash ordering differences in different systems/interpreters rather than being a PyPy-specific bug.

Revision history for this message
Jeremy Stanley (fungi) wrote :

I'm removing py3k-precise nodes from rax for the time being with https://review.openstack.org/79451 to get pypy jobs flowing consistently again while I dig deeper. I've opened a pip issue at https://github.com/pypa/pip/issues/1632 on the suspicion more answers lie there.

Jeremy Stanley (fungi)
Changed in openstack-ci:
status: Confirmed → In Progress
Jeremy Stanley (fungi)
Changed in openstack-ci:
importance: Critical → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to elastic-recheck (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/81015

Revision history for this message
Jeremy Stanley (fungi) wrote :

This started impacting hpcloud nodes over the weekend now, and seems coincidentally timed to the release of setuptools 3.3 (no idea whether that could be related to the new shift in behavior).

Revision history for this message
Jeremy Stanley (fungi) wrote :

For reasons as of yet unidentified, pip is able to upgrade setuptools to 3.1 safely but if it tries to go any higher, that's when we hit this bug. Projects wishing to work around it temporarily without disabling PyPy testing can apply something along the lines of http://paste.openstack.org/show/73701/ to their tox.ini files while a proper fix is identified (whether that be in pip, setuptools, the PyPy interpreter or somewhere else entirely).

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to elastic-recheck (master)

Reviewed: https://review.openstack.org/81015
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=1bc31d63deccc89aa227d1e8606335d78dcdf4cf
Submitter: Jenkins
Branch: master

commit 1bc31d63deccc89aa227d1e8606335d78dcdf4cf
Author: Joe Gordon <email address hidden>
Date: Mon Mar 17 09:46:49 2014 -0700

    Add fingerprint for bug 1290562

    Bug 1290562 is a setuptools/pypy bug

    Change-Id: Ia624769479b3d68e534fa101a04395b6809d98fb
    Related-Bug: #1290562

Changed in python-ceilometerclient:
assignee: nobody → Ildiko Vancsa (ildiko-vancsa)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-neutronclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/81413

Changed in python-neutronclient:
assignee: nobody → Akihiro Motoki (amotoki)
status: New → In Progress
Revision history for this message
Ildiko Vancsa (ildiko-vancsa) wrote :

Temporary fix added to python-ceilometerclient: https://review.openstack.org/81275

Changed in python-ceilometerclient:
status: New → Fix Committed
Revision history for this message
Chmouel Boudjnah (chmouel) wrote :

add a temporary fix as well for swiflicnet on https://review.openstack.org/#/c/81479/ (based on the one from ceilometerclient)

Revision history for this message
Noorul Islam K M (noorul) wrote :
Revision history for this message
Jeremy Stanley (fungi) wrote :

The underlying issue has been tracked down to incomplete PEP 3147 backport patches in Debian's pypy package, which are also inherited by the Ubuntu packages and those in the backport PPA we're using. The package maintainers are coordinating a fix currently, and once it's present in the PPA we can revert the current workarounds.

Revision history for this message
Jeremy Stanley (fungi) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-neutronclient (master)

Reviewed: https://review.openstack.org/81413
Committed: https://git.openstack.org/cgit/openstack/python-neutronclient/commit/?id=27d238d2954ff27b1c10f899479e54ed1ec00703
Submitter: Jenkins
Branch: master

commit 27d238d2954ff27b1c10f899479e54ed1ec00703
Author: Akihiro Motoki <email address hidden>
Date: Wed Mar 19 10:37:08 2014 +0900

    Work around pypy testing issue

    A recent release of setuptools appears to have introduced a bug that
    causes the pypy gate tests to fail. This applies a temporary workaround
    that should restore pypy testing while the root cause of the problem is
    researched.

    The work around is suggested in the comment in bug 1290562.

    Change-Id: Iac2ec604447b1638550219cd2e407daf188f8ede
    Partial-Bug: 1290562

Akihiro Motoki (amotoki)
Changed in python-neutronclient:
status: In Progress → Fix Committed
importance: Undecided → Medium
Revision history for this message
Adrian Otto (aotto) wrote :

Subsequent updates to setuptools have eliminated this issue.

Changed in solum:
status: New → Won't Fix
Changed in openstack-ci:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-neutronclient (master)

Reviewed: https://review.openstack.org/97075
Committed: https://git.openstack.org/cgit/openstack/python-neutronclient/commit/?id=8845ed254acbbd5ee1ff02688ea482996c58ef5d
Submitter: Jenkins
Branch: master

commit 8845ed254acbbd5ee1ff02688ea482996c58ef5d
Author: Alex Gaynor <email address hidden>
Date: Sat May 31 21:40:38 2014 -0700

    Removed now unnecesary workaround for PyPy

    Related-bug: #1290562

    Change-Id: I3fbfdcecfa19593729b1cb14685201207abc64a6

Revision history for this message
Sean Dague (sdague) wrote :

Dropped from ER tracking

Kyle Mestery (mestery)
Changed in python-neutronclient:
milestone: none → 2.3.5
Akihiro Motoki (amotoki)
Changed in python-neutronclient:
status: Fix Committed → Fix Released
Revision history for this message
Steven Hardy (shardy) wrote :

I've just seen a very similar failure to this for python-heatclient, raised https://bugs.launchpad.net/python-heatclient/+bug/1453095

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to project-config (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/181868

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to project-config (master)

Reviewed: https://review.openstack.org/181868
Committed: https://git.openstack.org/cgit/openstack-infra/project-config/commit/?id=da3f7003929388b3550dc059b37850dc35216752
Submitter: Jenkins
Branch: master

commit da3f7003929388b3550dc059b37850dc35216752
Author: Sean Dague <email address hidden>
Date: Mon May 11 07:53:05 2015 -0400

    make pypy non-voting across the board

    This is breaking at a fundamental level on all jobs. We should disable
    voting for pypy across the board to get things moving again, and
    consider removing this in the future.

    Consolidate the multiple definitions for pypy that would be created
    with this, and place the comments next to the correct logical blocks
    to make them less confusing.

    Related-Bug: #1290562

    Change-Id: If1cf7af0e950f9b7725e84127ee936bffd63ea6d

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to elastic-recheck (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/181919

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to elastic-recheck (master)

Reviewed: https://review.openstack.org/181919
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=94f94c9b3ad135213f2e8e370701db0c15a88fdd
Submitter: Jenkins
Branch: master

commit 94f94c9b3ad135213f2e8e370701db0c15a88fdd
Author: Matt Riedemann <email address hidden>
Date: Mon May 11 07:40:28 2015 -0700

    Add query for pypy job bug 1290562

    567 hits in 7 days, check and gate, all failures.

    Change-Id: I7cf583f53ff31f257cb07a6ee8d02b6d649364a7
    Related-Bug: #1290562

Changed in python-ceilometerclient:
milestone: none → 1.3.0
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.