pip in Python 3.4 virtualenvs cannot install using a proxy

Bug #1522500 reported by Dan Watkins
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
python-virtualenv (Ubuntu)
Won't Fix
High
Barry Warsaw

Bug Description

When using the version of python-virtualenv in trusty, I run in to problems installing from a proxy. This appears to be because the fix in https://github.com/shazow/urllib3/pull/385 (which was applied to trusty's python-urllib3 in 1.7.1-1ubuntu4) has not also been applied to the vendored version in this package. Details follow.

When I do the following:

virtualenv .venv
. ./.venv/bin/activate
http_proxy=http://squid.internal:3128/ https_proxy=http://squid.internal:3128/ HTTP_PROXY=http://squid.internal:3128/ HTTPS_PROXY=http://squid.internal:3128/ pip --proxy squid.internal:3128 install requests

I get an error like:

 Downloading/unpacking requests==2.8.1 (from -r requirements.txt (line 1))
   Cannot fetch index base URL https://pypi.python.org/simple/
   Could not find any downloads that satisfy the requirement requests==2.8.1 (from -r requirements.txt (line 1))
 Cleaning up...
 No distributions at all found for requests==2.8.1 (from -r requirements.txt (line 1))
 Storing debug log for failure in /var/lib/jenkins/.pip/pip.log

And pip.log contains:

Downloading/unpacking requests==2.8.1 (from -r requirements.txt (line 1))
  Getting page https://pypi.python.org/simple/requests/
  Could not fetch URL https://pypi.python.org/simple/requests/: connection error: hostname 'squid.internal' doesn't match either of 'www.python.org', 'python.org', 'pypi.python.org', 'docs.python.org', 'testpypi.python.org', 'bugs.python.org', 'wiki.python.org', 'hg.python.org', 'mail.python.org', 'packaging.python.org', 'pythonhosted.org', 'www.pythonhosted.org', 'test.pythonhosted.org', 'us.pycon.org', 'id.python.org'
  Will skip URL https://pypi.python.org/simple/requests/ when looking for download links for requests==2.8.1 (from -r requirements.txt (line 1))
  Getting page https://pypi.python.org/simple/
  Could not fetch URL https://pypi.python.org/simple/: connection error: hostname 'squid.internal' doesn't match either of 'www.python.org', 'python.org', 'pypi.python.org', 'docs.python.org', 'testpypi.python.org', 'bugs.python.org', 'wiki.python.org', 'hg.python.org', 'mail.python.org', 'packaging.python.org', 'pythonhosted.org', 'www.pythonhosted.org', 'test.pythonhosted.org', 'us.pycon.org', 'id.python.org'
  Will skip URL https://pypi.python.org/simple/ when looking for download links for requests==2.8.1 (from -r requirements.txt (line 1))
  Cannot fetch index base URL https://pypi.python.org/simple/
  URLs to search for versions for requests==2.8.1 (from -r requirements.txt (line 1)):
  * https://pypi.python.org/simple/requests/2.8.1
  * https://pypi.python.org/simple/requests/
  Getting page https://pypi.python.org/simple/requests/2.8.1
  Could not fetch URL https://pypi.python.org/simple/requests/2.8.1: connection error: hostname 'squid.internal' doesn't match either of 'www.python.org', 'python.org', 'pypi.python.org', 'docs.python.org', 'testpypi.python.org', 'bugs.python.org', 'wiki.python.org', 'hg.python.org', 'mail.python.org', 'packaging.python.org', 'pythonhosted.org', 'www.pythonhosted.org', 'test.pythonhosted.org', 'us.pycon.org', 'id.python.org'
  Will skip URL https://pypi.python.org/simple/requests/2.8.1 when looking for download links for requests==2.8.1 (from -r requirements.txt (line 1))
  Getting page https://pypi.python.org/simple/requests/
  Could not fetch URL https://pypi.python.org/simple/requests/: connection error: hostname 'squid.internal' doesn't match either of 'www.python.org', 'python.org', 'pypi.python.org', 'docs.python.org', 'testpypi.python.org', 'bugs.python.org', 'wiki.python.org', 'hg.python.org', 'mail.python.org', 'packaging.python.org', 'pythonhosted.org', 'www.pythonhosted.org', 'test.pythonhosted.org', 'us.pycon.org', 'id.python.org'
  Will skip URL https://pypi.python.org/simple/requests/ when looking for download links for requests==2.8.1 (from -r requirements.txt (line 1))
  Could not find any downloads that satisfy the requirement requests==2.8.1 (from -r requirements.txt (line 1))
Cleaning up...
  Removing temporary dir /var/lib/jenkins/slaves/jenkins-slave-0/workspace/16.04-Trigger/.venv/build...
No distributions at all found for requests==2.8.1 (from -r requirements.txt (line 1))
Exception information:
Traceback (most recent call last):
  File "/var/lib/jenkins/slaves/jenkins-slave-0/workspace/16.04-Trigger/.venv/lib/python3.4/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/var/lib/jenkins/slaves/jenkins-slave-0/workspace/16.04-Trigger/.venv/lib/python3.4/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/var/lib/jenkins/slaves/jenkins-slave-0/workspace/16.04-Trigger/.venv/lib/python3.4/site-packages/pip/req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/var/lib/jenkins/slaves/jenkins-slave-0/workspace/16.04-Trigger/.venv/lib/python3.4/site-packages/pip/index.py", line 277, in find_requirement
    raise DistributionNotFound('No distributions at all found for %s' % req)
pip.exceptions.DistributionNotFound: No distributions at all found for requests==2.8.1 (from -r requirements.txt (line 1))

Barry Warsaw (barry)
Changed in python-virtualenv (Ubuntu):
assignee: nobody → Barry Warsaw (barry)
status: New → In Progress
importance: Undecided → High
Revision history for this message
Barry Warsaw (barry) wrote :

To be clear `virtualenv <dir>` will create a Python 2.7 virtualenv, and this works fine.

Use `virtualenv -p python3.4 <dir>` to trigger the described failure, which I've been able to reproduce.

Revision history for this message
Barry Warsaw (barry) wrote :

Here's what's going on. Keep in mind, that post-Trusty this was all sorted out but backporting the fixes to Trusty will not be easy or quick.

In Trusty, virtualenv vendorizes pip, which vendorizes requests, which vendorizes urllib3. It's the three-layers down vendored urllib3 that would need to be fixed, but doing that is not easy because of the way this whole stack interacts. I think we'd essentially have to backport everything to either devendorize everything, or build the pip.whl back up using a patched urllib3 all the way back up to virtualenv. It's a massive amount of work.

There's good news though: the virtual environment support built into Python 3.4 already has all the necessary fixes, so I would highly recommend using that instead. First, apt-get install python3.4-venv (for some reason there's no virtual package python3-venv in Trusty as there is in Vivid and beyond, but no matter, the python3.4-venv package does the trick).

Then: `python3 -m venv <dir>`. Source that venv's activate script, set your proxies up, and everything should install just fine through squid.

Barry Warsaw (barry)
Changed in python-virtualenv (Ubuntu):
status: In Progress → Won't Fix
Revision history for this message
Dan Watkins (oddbloke) wrote :

I have confirmed that this workaround does, indeed, work around the problem. Thanks!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.