pip-requires and virtualenv don't play together nicely

Note that a the installation won't work if the developer does not have python's development headers, since M2Crypto builds the extension manually and must link with Python.h.

Not sure how we can enforce this with the pip-requires file? Open to suggestions.

Yeah i was noting this. On ubuntu you can do:
apt-get build-dep python-m2crypto
Should we check and run the apt-get command?

Vish

On Tue, Aug 10, 2010 at 9:53 AM, Jay Pipes <email address hidden>wrote:

> Note that a the installation won't work if the developer does not have
> python's development headers, since M2Crypto builds the extension
> manually and must link with Python.h.
>
> Not sure how we can enforce this with the pip-requires file? Open to
> suggestions.
>
> --
> pip-requires and virtualenv don't play together nicely
> https://bugs.launchpad.net/bugs/615906
> You received this bug notification because you are a member of Nova
> Bugs, which is subscribed to OpenStack Compute (nova).
>
> Status in OpenStack Compute (Nova): Fix Committed
>
> Bug description:
> OK, so I'm familiar with virtualenv, and have been developing Nova with
> virtualenv for a while now, with no problems if I use the instructions I put
> up on my blog here:
>
> http://www.joinfu.com/2010/07/developing-nova-on-linux-getting-started/
>
> However, if I try to use the "automated" virtualenv/run_tests.sh stuff,
> everything bombs, and IMHO, it bombs unnecessarily.
>
> I'm on Linux, Ubuntu Lucid, but it really shouldn't matter *what* I have
> installed locally if we use virtualenv properly. Let me show you what
> happens when I try to do things "the automated" way using run_tests.sh and
> /tools/install_virtualenv.py.
>
> jpipes@serialcoder:~/repos/nova/pylint-nova-objectstore-handler$
> ./run_tests.sh
> You need to install the Nova virtualenv before you can run this.
>
> Please run tools/install_venv.py
> jpipes@serialcoder:~/repos/nova/pylint-nova-objectstore-handler$ python
> tools/install_venv.py
> Checking for pip... done.
> Checking for virtualenv... done.
> Creating venv... done.
> Installing dependencies with pip (this can take a while)...
> Exception:
> Traceback (most recent call last):
> File "/usr/lib/python2.6/dist-packages/pip.py", line 252, in main
> self.run(options, args)
> File "/usr/lib/python2.6/dist-packages/pip.py", line 408, in run
> requirement_set.install_files(finder, force_root_egg_info=self.bundle)
> File "/usr/lib/python2.6/dist-packages/pip.py", line 1721, in
> install_files
> if req_to_install.check_if_exists():
> File "/usr/lib/python2.6/dist-packages/pip.py", line 1583, in
> check_if_exists
> dist = pkg_resources.get_distribution(self.req)
> File "/usr/share/pyshared/pkg_resources.py", line 292, in
> get_distribution
> if isinstance(dist,Requirement): dist = get_provider(dist)
> File "/usr/share/pyshared/pkg_resources.py", line 176, in get_provider
> return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
> File "/usr/share/pyshared/pkg_resources.py", line 436, in find
> raise VersionConflict(dist,req) # XXX add more info
> VersionConflict: (anyjson 0.2.2 (/usr/share/pyshared),
> Requirement.parse('anyjson==0.2.4'))
>
> Storing complete log in ./pip-log.txt
> Command "pip install -E .nova-venv -r tools/pip-requires" failed.
> None
>
> ===
>
> This is just outright annoying. The whole purpose of a virtualenv is to
> install packages into a virtual environment that d...

So...I was trying to keep Andy's original intention of being platform-agnostic in the script, so I only rely on easy_install.

Lemme ask Monty (or Andy, if you know!) if setup.py has some ability to do a C++-like configure.ac for checking non-Python build dependencies like this...

I am late to this thread but I don't think this issue is valid.

I don't understand how you are getting the error you are getting. Either you are using a clean virtualenv and the only version of anyjson available for download is 0.2.4 or you are not using a clean virtualenv.

In the first case it means we should change the pip-requires file to be more lenient on the version of anyjson, in fact we don't even require it some other library does.

In the second case we just need to make sure we are passing the right flags to make a clean virtual env.

Towards the patch that already appears "Merged" the exact opposite patch to change the behavior form doing what you are already doing is how we got to this point, so that patch should be reverted pending discussion with the people who reviewed and discussed the original patch (i think myself, monty and soren). The main goal was not to install anything on anybody's system without them agreeing to it first, and additionally any packages that need to be installed globally probably need to be run with root so the script should choke if it tried to install virtualenv without being run with sudo.

As for installing pip in the virtualenv itself that seems fine, though I don't feel like it has much impact.

As for checking the build stuff for m2crypto I don't think there is much of a way around telling them it is somewhat tricky, but we can certainly check if they are linux and if they are check with apt-get and tell them what to do, `uname` is usually enough to determine their platform or we can `which` apt-get

Going to put in a follow-up patch to remove the automatic installation of virtualenv. Per IRC convo with termie:

<jaypipes> termie: so...this bug: https://bugs.launchpad.net/nova/+bug/610402
<jaypipes> termie: that is the original one you are referring to, right?
<jaypipes> termie: in regards to my patch on installing into virtualenv?
<termie> jaypipes: no, not really
<termie> jaypipes: so, i first added this makefile that did this stuff
<termie> soren had issue with it because it was installing things on the system
<termie> the decision was to move the installation to a separate script
<termie> then the makefile changed to suit hudson
<termie> then the makefil was removed to suit debian
<jaypipes> termie: ah, ok, which was the tools/with_virt.sh and install_venv?
<termie> and the test stuff put into run_test.ssh
<termie> jaypipes: correcy
<jaypipes> gotcha.
<termie> the main goal of the whole thing is just to try to make everybody use the same environment
<termie> so that bug reports are limited to actual bugs rather than library mismatches
<termie> and to make it easier for people to run the tests so that they actually do it
<jaypipes> termie: so, the newer bug (615906) was due to, apparently, when running install_venv, the pip -E was apparently *not* respecting the --no-site-packages... or at least, that's what it seemed to me. There should be no need to have *any* version of anyjson on my local system for run_tests.sh and install_venv.py to succeed (since pip should just install it into the venv), but that's not what happened.
* jaypipes thinks that is hard to follow...but that's the story :)
<termie> jaypipes: right, which is what i was guessing in my statement that it was not a clean virtualenv
<termie> so i think the bugfix is to make sure it is a clean virtualenv
<jaypipes> termie: but it was :)
<termie> jaypipes: then fix the bug ;)
<jaypipes> termie: oh, are you saying that the fix was to rm -rf the virtualenv?
<termie> no, i am saying the fix is to make sure that you are getting the version of anyjson that is in the virtualenv
* jaypipes really not understanding...
<termie> since that is the main purpose of vitualenv and it tends to work quite well at it i am assumign somewhere in here we have some user error and my code was doing it wrong
<termie> if pip is not respecting the virtualenv when it is checking dependencies we should find a way to make sure it does
<termie> because it certainly seems to every time i use it
<termie> otherwise it wouldn't be installing things i already have on my system
<jaypipes> termie: I can only tell you what happened...
<jaypipes> termie: the only solution I found was to adapt the install_venv script to call tools/with_venv.sh pip -E blah blah instead of just pip -E blah blah...
<termie> jaypipes: then perhaps that is a fine solution
<jaypipes> termie: which, reading the docs for pip seemed redundant, but unfortunately, that's all that worked. :(
<termie> jaypipes: but the patch you have is not limited to that
<jaypipes> termie: the only other thing the patch does is automatically install virtualenv instead of returning an error message saying virtualenv is required...
<jaypipes> termie: when run...

On 08/11/2010 08:32 AM, Jay Pipes wrote:
> So...I was trying to keep Andy's original intention of being platform-
> agnostic in the script, so I only rely on easy_install.
>
> Lemme ask Monty (or Andy, if you know!) if setup.py has some ability to
> do a C++-like configure.ac for checking non-Python build dependencies
> like this...

Not really - it's the main deficiency in using easy_install/etc. rather
than system stuff.

OTOH, it _is_ python, so one can write anything, but it's not a feature
of the system and I'm not sure I believe we should put it in there.

I think it may just have to be ok for there to be a step or two someone
has to do (such as apt-get install) to get some of the components. They
aren't going to be installed into the virtualenv anyway...

Monty

Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is prohibited.
If you receive this transmission in error, please notify us immediately by e-mail
at <email address hidden>, and delete the original message.
Your cooperation is appreciated.