"keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty." in test_cinderclient_unsupported_v1

Bug #1654393 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Matt Riedemann

Bug Description

This just started failing today:

http://logs.openstack.org/18/417118/1/check/gate-nova-python27-db-ubuntu-xenial/0c0c8c1/console.html#_2017-01-05_17_52_16_199447

2017-01-05 17:52:16.199447 | nova.tests.unit.test_cinder.CinderV1TestCase.test_cinderclient_unsupported_v1
2017-01-05 17:52:16.199481 | -----------------------------------------------------------------------------
2017-01-05 17:52:16.199488 |
2017-01-05 17:52:16.199498 | Captured traceback:
2017-01-05 17:52:16.199510 | ~~~~~~~~~~~~~~~~~~~
2017-01-05 17:52:16.199525 | Traceback (most recent call last):
2017-01-05 17:52:16.199568 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
2017-01-05 17:52:16.199585 | return func(*args, **keywargs)
2017-01-05 17:52:16.199612 | File "nova/tests/unit/test_cinder.py", line 134, in test_cinderclient_unsupported_v1
2017-01-05 17:52:16.199627 | cinder.cinderclient, ctxt)
2017-01-05 17:52:16.199684 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 485, in assertRaises
2017-01-05 17:52:16.199701 | self.assertThat(our_callable, matcher)
2017-01-05 17:52:16.199768 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 496, in assertThat
2017-01-05 17:52:16.199794 | mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
2017-01-05 17:52:16.199839 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 547, in _matchHelper
2017-01-05 17:52:16.199854 | mismatch = matcher.match(matchee)
2017-01-05 17:52:16.199899 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
2017-01-05 17:52:16.199919 | mismatch = self.exception_matcher.match(exc_info)
2017-01-05 17:52:16.199975 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
2017-01-05 17:52:16.199992 | mismatch = matcher.match(matchee)
2017-01-05 17:52:16.200033 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 475, in match
2017-01-05 17:52:16.200045 | reraise(*matchee)
2017-01-05 17:52:16.200088 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
2017-01-05 17:52:16.200100 | result = matchee()
2017-01-05 17:52:16.200142 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 1049, in __call__
2017-01-05 17:52:16.200162 | return self._callable_object(*self._args, **self._kwargs)
2017-01-05 17:52:16.200181 | File "nova/volume/cinder.py", line 80, in cinderclient
2017-01-05 17:52:16.200203 | url = _SESSION.get_endpoint(auth, **service_parameters)
2017-01-05 17:52:16.200247 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/keystoneauth1/session.py", line 808, in get_endpoint
2017-01-05 17:52:16.200263 | return auth.get_endpoint(self, **kwargs)
2017-01-05 17:52:16.200281 | File "nova/context.py", line 61, in get_endpoint
2017-01-05 17:52:16.200294 | region_name=region_name)
2017-01-05 17:52:16.200391 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-01-05 17:52:16.200420 | return wrapped(*args, **kwargs)
2017-01-05 17:52:16.200467 | File "/home/jenkins/workspace/gate-nova-python27-db-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 190, in url_for
2017-01-05 17:52:16.200490 | raise exceptions.EmptyCatalog('The service catalog is empty.')
2017-01-05 17:52:16.200527 | keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty.

It's a race in a test introduced in https://review.openstack.org/#/c/409904/.

I suspect it's a race in a global mock.

Tags: testing
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/417206

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
status: Confirmed → In Progress
Revision history for this message
Matthew Booth (mbooth-9) wrote :

A minimal order to deterministically reproduce this issue is:

nova.tests.unit.compute.test_compute_mgr.ComputeManagerUnitTestCase.test_shutdown_instance_versioned_notifications
nova.tests.unit.test_cinder.CinderV1TestCase.test_cinderclient_unsupported_v1

I'm currently looking for a root cause in ComputeManagerUnitTestCase.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/417408

Changed in nova:
assignee: Matt Riedemann (mriedem) → Matthew Booth (mbooth-9)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Matthew Booth (<email address hidden>) on branch: master
Review: https://review.openstack.org/417408
Reason: We're going with with an alternate approach.

Matt Riedemann (mriedem)
Changed in nova:
assignee: Matthew Booth (mbooth-9) → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.0.0b3

This issue was fixed in the openstack/nova 15.0.0.0b3 development milestone.

Revision history for this message
Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

Changed in nova:
status: In Progress → Confirmed
assignee: Matt Riedemann (mriedem) → nobody
Matt Riedemann (mriedem)
Changed in nova:
status: Confirmed → Fix Released
assignee: nobody → Matt Riedemann (mriedem)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.