keystonemiddleware

DuplicateOptError intermittently breaks UT

Bug #1494327 reported by Matthew Edmonds
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
keystonemiddleware
Fix Released
Critical
Brant Knudson
keystonemiddleware 2.3.0 "2.3.0"

Bug Description

keystonemiddleware UTs sometimes (pretty often, but not consistently) fail with a DuplicateOptError as seen here: http://logs.openstack.org/73/217373/6/check/gate-keystonemiddleware-python34/03e20d0/console.html

2015-09-09 23:26:24.494 | FAIL: keystonemiddleware.tests.unit.auth_token.test_user_auth_plugin.V2UserPluginTests.test_with_service_information
2015-09-09 23:26:24.494 | tags: worker-3
2015-09-09 23:26:24.495 | ----------------------------------------------------------------------
2015-09-09 23:26:24.495 | Empty attachments:
2015-09-09 23:26:24.495 | pythonlogging:''
2015-09-09 23:26:24.495 | stderr
2015-09-09 23:26:24.495 | stdout
2015-09-09 23:26:24.495 |
2015-09-09 23:26:24.495 | Traceback (most recent call last):
2015-09-09 23:26:24.496 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/keystonemiddleware/tests/unit/auth_token/test_user_auth_plugin.py", line 95, in setUp
2015-09-09 23:26:24.496 | tenant_id=self.service_token.tenant_id)
2015-09-09 23:26:24.497 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/keystonemiddleware/tests/unit/auth_token/test_user_auth_plugin.py", line 32, in configure_middleware
2015-09-09 23:26:24.497 | self.cfg.register_opts(opts, group=group)
2015-09-09 23:26:24.497 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/fixture.py", line 105, in register_opts
2015-09-09 23:26:24.497 | self.register_opt(opt, group=group)
2015-09-09 23:26:24.497 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/fixture.py", line 93, in register_opt
2015-09-09 23:26:24.497 | self.conf.register_opt(opt, group=group)
2015-09-09 23:26:24.497 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/cfg.py", line 1824, in __inner
2015-09-09 23:26:24.498 | result = f(self, *args, **kwargs)
2015-09-09 23:26:24.498 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/cfg.py", line 1967, in register_opt
2015-09-09 23:26:24.498 | return group._register_opt(opt, cli)
2015-09-09 23:26:24.498 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/cfg.py", line 1345, in _register_opt
2015-09-09 23:26:24.498 | if _is_opt_registered(self._opts, opt):
2015-09-09 23:26:24.498 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/cfg.py", line 574, in _is_opt_registered
2015-09-09 23:26:24.499 | raise DuplicateOptError(opt.name)
2015-09-09 23:26:24.501 | oslo_config.cfg.DuplicateOptError: duplicate option: user-name

How to reproduce:
Just run `tox -e py27` in the keystonemiddleware code base serially, you will hit the failure as this.

This is referenced with this commit (https://review.openstack.org/#/c/216579) but it is not the root cause, we will still see the DuplicateOptError with user-id option but it's relatively not such frequently.

See original description
Dolph Mathews (dolph)
Changed in keystonemiddleware:
status: New → Confirmed
importance: Undecided → High
Lin Hua Cheng (lin-hua-cheng)
Changed in keystonemiddleware:
assignee: nobody → Lin Hua Cheng (lin-hua-cheng)
Revision history for this message
Lin Hua Cheng (lin-hua-cheng) wrote :

Getting close..

Isolated the test failure in : https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/tests/unit/auth_token/test_auth_token_middleware.py#L691-L702

Revision history for this message
Lin Hua Cheng (lin-hua-cheng) wrote :

if these lines are removed, the tests passes consistently: https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/tests/unit/auth_token/test_auth_token_middleware.py#L691-L702

Revision history for this message
Dave Chen (wei-d-chen) wrote :

yep, I found the failure is caused by these testcase too, and asked in the oslo channel but got no answer.

Revision history for this message
Dave Chen (wei-d-chen) wrote :

And this is primarly caused by the `keystone_authtoken` group is not cleared after the testcase, then when `user-name` options is registered again in other testcase, the issue happened, there is no way in the oslo-config that could clear the group options. I am wondering whether this is a bug in oslo-config.

OpenStack Infra (hudson-openstack)
Changed in keystonemiddleware:
assignee: Lin Hua Cheng (lin-hua-cheng) → Dave Chen (wei-d-chen)
status: Confirmed → In Progress
Dave Chen (wei-d-chen)
description: updated
Revision history for this message
Brant Knudson (blk-u) wrote :

I'm able to recreate this consistently:

$ echo "keystonemiddleware.tests.unit.auth_token.test_auth_token_middleware.GeneralAuthTokenMiddlewareTest.test_auth_region_name" > my-list
$ echo "keystonemiddleware.tests.unit.auth_token.test_user_auth_plugin.V2UserPluginTests.test_user_information" >> my-list
$ .tox/py27/bin/python -m testtools.run discover --load-list my-list

These aren't the only 2 tests that will cause the 2nd test to fail. Could also use keystonemiddleware.tests.unit.auth_token.test_auth_token_middleware.GeneralAuthTokenMiddlewareTest.test_auth_region_name . I think it's actually any test that creates and AuthProtocol.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystonemiddleware (master)

Fix proposed to branch: master
Review: https://review.openstack.org/223338

Changed in keystonemiddleware:
assignee: Dave Chen (wei-d-chen) → Brant Knudson (blk-u)
Brant Knudson (blk-u)
Changed in keystonemiddleware:
assignee: Brant Knudson (blk-u) → Dave Chen (wei-d-chen)
Revision history for this message
Matthew Edmonds (edmondsw) wrote :

Dave has also proposed https://review.openstack.org/#/c/222919/ that I don't see linked here.

OpenStack Infra (hudson-openstack)
Changed in keystonemiddleware:
assignee: Dave Chen (wei-d-chen) → Brant Knudson (blk-u)
Brant Knudson (blk-u)
Changed in keystonemiddleware:
assignee: Brant Knudson (blk-u) → Dave Chen (wei-d-chen)
OpenStack Infra (hudson-openstack)
Changed in keystonemiddleware:
assignee: Dave Chen (wei-d-chen) → Brant Knudson (blk-u)
Brant Knudson (blk-u)
Changed in keystonemiddleware:
assignee: Brant Knudson (blk-u) → Dave Chen (wei-d-chen)
OpenStack Infra (hudson-openstack)
Changed in keystonemiddleware:
assignee: Dave Chen (wei-d-chen) → Brant Knudson (blk-u)
Revision history for this message
Brant Knudson (blk-u) wrote :

This is preventing us from merging changes (such as requirements updates) in keystonemiddleware, and we've got to make these changes for the liberty freeze, so marking as critical.

Changed in keystonemiddleware:
importance: High → Critical
assignee: Brant Knudson (blk-u) → Dave Chen (wei-d-chen)
Revision history for this message
Dave Chen (wei-d-chen) wrote :

Change the assignee to Brant Knudson since he has the patch will properly the right fix.

Changed in keystonemiddleware:
assignee: Dave Chen (wei-d-chen) → Brant Knudson (blk-u)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystonemiddleware (master)

Reviewed: https://review.openstack.org/223338
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=5352ff068dc0466b3a376e802512c2065c130455
Submitter: Jenkins
Branch: master

commit 5352ff068dc0466b3a376e802512c2065c130455
Author: Brant Knudson <email address hidden>
Date: Mon Sep 14 18:18:32 2015 -0500

    auth_token tests use clean config

    The auth_token tests were failing randomly (depending on which tests
    were run together) because the global config had options left in it
    by other tests (options were added as part of auth plugin
    initialization and oslo.config provides no way to remove registered
    options).

    This change makes it so that auth_token gets a fresh config for
    every test (other than one test that requires using the global CONF
    object).

    Closes-Bug: 1494327
    Change-Id: I647f7fade01e2a619f4a5d12a4e71897423469c1

Changed in keystonemiddleware:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on keystonemiddleware (master)

Change abandoned by Brant Knudson (<email address hidden>) on branch: master
Review: https://review.openstack.org/222919
Reason: With https://review.openstack.org/#/c/223338/ , all the options are reset after each test, so no need for this.

Davanum Srinivas (DIMS) (dims-v)
Changed in keystonemiddleware:
milestone: none → 2.3.0
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.