Network is unreachable errno string reported in various ways killing pip and git operations gate/check jobs

Bug #1275994 reported by Peter Portante
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Core Infrastructure
Expired
Medium
Unassigned

Bug Description

The "Network is unreachable" errno string reported in various ways killing pip and git operations gate/check jobs. It seems to be reported mostly with pip, accessing pypi.openstack.org, sometimes with git.openstack.org, and less often with others.

One example of how it manifests itself is killing pip installs (see: http://logs.openstack.org/91/70691/2/check/check-devstack-dsvm-cells/9dd947b/console.html):

2014-02-03 11:09:07.482 | 2014-02-03 11:09:07 + sudo PIP_DOWNLOAD_CACHE=/var/cache/pip HTTP_PROXY= HTTPS_PROXY= NO_PROXY= /usr/local/bin/pip install --build=/tmp/pip-build.GGbB5 -e /opt/stack/new/horizon
2014-02-03 11:09:07.482 | 2014-02-03 11:09:07 Obtaining file:///opt/stack/new/horizon
2014-02-03 11:09:07.483 | 2014-02-03 11:09:07 Running setup.py egg_info for package from file:///opt/stack/new/horizon
2014-02-03 11:09:07.483 | 2014-02-03 11:09:07 [pbr] Processing SOURCES.txt
2014-02-03 11:09:07.483 | 2014-02-03 11:09:07 warning: LocalManifestMaker: standard file '-c' not found
2014-02-03 11:09:07.484 | 2014-02-03 11:09:07
2014-02-03 11:09:07.514 | 2014-02-03 11:09:07 [pbr] In git context, generating filelist from git
2014-02-03 11:09:07.514 | 2014-02-03 11:09:07 warning: no files found matching 'AUTHORS'
2014-02-03 11:09:07.515 | 2014-02-03 11:09:07 warning: no files found matching 'ChangeLog'
2014-02-03 11:09:07.515 | 2014-02-03 11:09:07 warning: no previously-included files matching '*.pyc' found anywhere in distribution
2014-02-03 11:09:07.516 | 2014-02-03 11:09:07 warning: no files found matching '*.css' under directory 'doc'
2014-02-03 11:09:07.516 | 2014-02-03 11:09:07 warning: no files found matching '*.js' under directory 'doc'
2014-02-03 11:09:07.516 | 2014-02-03 11:09:07 warning: no files found matching '*.html' under directory 'doc'
2014-02-03 11:09:07.517 | 2014-02-03 11:09:07 warning: no files found matching '*.conf' under directory 'doc'
2014-02-03 11:09:07.517 | 2014-02-03 11:09:07 warning: no files found matching '*.jpg' under directory 'doc'
2014-02-03 11:09:07.517 | 2014-02-03 11:09:07 warning: no files found matching '*.gif' under directory 'doc'
2014-02-03 11:09:07.518 | 2014-02-03 11:09:07 warning: no files found matching '*.png' under directory 'doc'
2014-02-03 11:09:07.518 | 2014-02-03 11:09:07 warning: no files found matching '*.css_t' under directory 'doc'
2014-02-03 11:09:07.518 | 2014-02-03 11:09:07 warning: no files found matching '*.csv' under directory 'horizon'
2014-02-03 11:09:07.519 | 2014-02-03 11:09:07 warning: no files found matching '*.template' under directory 'horizon'
2014-02-03 11:09:07.519 | 2014-02-03 11:09:07 warning: no files found matching '*.js' under directory 'openstack_dashboard'
2014-02-03 11:09:07.519 | 2014-02-03 11:09:07 warning: no files found matching 'AUTHORS'
2014-02-03 11:09:07.520 | 2014-02-03 11:09:07 warning: no files found matching 'ChangeLog'
2014-02-03 11:09:07.520 | 2014-02-03 11:09:07 warning: no files found matching 'doc/source/_templates/.placeholder'
2014-02-03 11:09:07.520 | 2014-02-03 11:09:07 warning: no previously-included files found matching 'openstack_dashboard/local/local_settings.py'
2014-02-03 11:09:07.521 | 2014-02-03 11:09:07 Requirement already satisfied (use --upgrade to upgrade): pbr>=0.5.21,<1.0 in /opt/stack/new/pbr (from horizon==2014.1.dev47.gc1fbaa8)
2014-02-03 11:09:07.521 | 2014-02-03 11:09:07 Downloading/unpacking Django>=1.4,<1.6 (from horizon==2014.1.dev47.gc1fbaa8)
2014-02-03 11:09:07.521 | 2014-02-03 11:09:07 Error <urlopen error [Errno 101] Network is unreachable> while getting http://pypi.openstack.org/openstack/Django/Django-1.5.5.tar.gz (from http://pypi.openstack.org/openstack/Django/)
2014-02-03 11:09:07.522 | 2014-02-03 11:09:07 Cleaning up...
2014-02-03 11:09:07.522 | 2014-02-03 11:09:07 Exception:
2014-02-03 11:09:07.522 | 2014-02-03 11:09:07 Traceback (most recent call last):
2014-02-03 11:09:07.551 | 2014-02-03 11:09:07 File "/usr/local/lib/python2.7/dist-packages/pip-1.4.1-py2.7.egg/pip/basecommand.py", line 134, in main
2014-02-03 11:09:07.551 | 2014-02-03 11:09:07 status = self.run(options, args)
2014-02-03 11:09:07.804 | 2014-02-03 11:09:07 File "/usr/local/lib/python2.7/dist-packages/pip-1.4.1-py2.7.egg/pip/commands/install.py", line 236, in run
2014-02-03 11:09:07.806 | 2014-02-03 11:09:07 requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
2014-02-03 11:09:07.808 | 2014-02-03 11:09:07 File "/usr/local/lib/python2.7/dist-packages/pip-1.4.1-py2.7.egg/pip/req.py", line 1092, in prepare_files
2014-02-03 11:09:07.809 | 2014-02-03 11:09:07 self.unpack_url(url, location, self.is_download)
2014-02-03 11:09:07.811 | 2014-02-03 11:09:07 File "/usr/local/lib/python2.7/dist-packages/pip-1.4.1-py2.7.egg/pip/req.py", line 1238, in unpack_url
2014-02-03 11:09:07.847 | 2014-02-03 11:09:07 retval = unpack_http_url(link, location, self.download_cache, self.download_dir)
2014-02-03 11:09:07.848 | 2014-02-03 11:09:07 File "/usr/local/lib/python2.7/dist-packages/pip-1.4.1-py2.7.egg/pip/download.py", line 602, in unpack_http_url
2014-02-03 11:09:07.849 | 2014-02-03 11:09:07 resp = _get_response_from_url(target_url, link)
2014-02-03 11:09:07.849 | 2014-02-03 11:09:07 File "/usr/local/lib/python2.7/dist-packages/pip-1.4.1-py2.7.egg/pip/download.py", line 638, in _get_response_from_url
2014-02-03 11:09:07.849 | 2014-02-03 11:09:07 resp = urlopen(target_url)
2014-02-03 11:09:07.850 | 2014-02-03 11:09:07 File "/usr/local/lib/python2.7/dist-packages/pip-1.4.1-py2.7.egg/pip/download.py", line 176, in __call__
2014-02-03 11:09:07.850 | 2014-02-03 11:09:07 response = self.get_opener(scheme=scheme).open(url)
2014-02-03 11:09:07.851 | 2014-02-03 11:09:07 File "/usr/lib/python2.7/urllib2.py", line 400, in open
2014-02-03 11:09:07.851 | 2014-02-03 11:09:07 response = self._open(req, data)
2014-02-03 11:09:07.852 | 2014-02-03 11:09:07 File "/usr/lib/python2.7/urllib2.py", line 418, in _open
2014-02-03 11:09:07.852 | 2014-02-03 11:09:07 '_open', req)
2014-02-03 11:09:07.853 | 2014-02-03 11:09:07 File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
2014-02-03 11:09:07.853 | 2014-02-03 11:09:07 result = func(*args)
2014-02-03 11:09:07.853 | 2014-02-03 11:09:07 File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
2014-02-03 11:09:07.854 | 2014-02-03 11:09:07 return self.do_open(httplib.HTTPConnection, req)
2014-02-03 11:09:07.854 | 2014-02-03 11:09:07 File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
2014-02-03 11:09:07.854 | 2014-02-03 11:09:07 raise URLError(err)
2014-02-03 11:09:07.855 | 2014-02-03 11:09:07 URLError: <urlopen error [Errno 101] Network is unreachable>

Bug 1266616 appears to be a subset of these failures.

Tags: gate-failure
Revision history for this message
Peter Portante (peter-a-portante) wrote :

This might share the same root cause with bug 1270710.

Revision history for this message
James E. Blair (corvus) wrote :

I don't see any obvious service degradation on pypi.o.o that would cause this; and the fact that it also affects git operations suggests it may be network or cloud-provider related. It may be worthwhile to note which cloud providers (eg rax-iad or hpcloud-az1) are most affected by this error.

Leaving this open to collect more data.

Changed in openstack-ci:
status: New → Incomplete
importance: Undecided → Medium
James E. Blair (corvus)
tags: added: gate-failure
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack Core Infrastructure because there has been no activity for 60 days.]

Changed in openstack-ci:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.