Loosing connection to keystone during CI

Bug #1337230 reported by Derek Higgins
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Clint Byrum

Bug Description

From http://logs.openstack.org/95/103995/2/check-tripleo/check-tripleo-overcloud-f20/2dd0ac6/console.html

+ keystone role-create --name=swiftoperator
+----------+----------------------------------+
| Property | Value |
+----------+----------------------------------+
| id | 0a2866ef76694e8cb03d3d7cec9fc35a |
| name | swiftoperator |
+----------+----------------------------------+
+ keystone role-create --name=ResellerAdmin
Authorization Failed: Unable to establish connection to http://192.0.2.3:5000/v2.0/tokens
+ get_state_from_hosts

Occurrences seem to have increased recently (38 in the last 7 days)

Tags: ci
Derek Higgins (derekh)
tags: added: ci
Revision history for this message
Derek Higgins (derekh) wrote :

It looks like o-c-c continues to run even after it completed successfully

[2014-07-03 08:39:08,935] (os-refresh-config) [INFO] Completed phase migration
INFO:os-refresh-config:Completed phase migration
2014-07-03 08:39:10.373 784 WARNING os_collect_config.heat [-] No auth_url configured.
2014-07-03 08:39:10.375 784 WARNING os-collect-config [-] Source [heat] Unavailable.
WARNING:root:Base directory /opt/stack/os-config-refresh is deprecated. The recommended base directory is /usr/libexec/os-refresh-config
[2014-07-03 08:39:10,412] (os-refresh-config) [INFO] Starting phase pre-configure
INFO:os-refresh-config:Starting phase pre-configure

A patch went in the coincides with this starting to become a problem
https://review.openstack.org/#/c/97703/
"Compute all hosts matrix once"

Which changes the way the metadata if calculated, I suspect it may be causing slight changes in the metadata over time re-triggering o-c-c, it had to be rechecked 10 times which may have been an indication of the
 problem, especially since a comment before one of the rechecks reads

"""
very strange race/failure:
2014-06-12 09:44:53.393 | Service ec2 created 2014-06-12 09:44:55.624 | Authorization Failed: Unable to establish connection to http://192.0.2.3:5000/v2.0/tokens 2014-06-12 09:44:55.895 | Authorization Failed: Unable to establish connection to http://192.0.2.3:5000/v2.0/tokens 2014-06-12 09:44:56.123 | usage: keystone user-role-add --user <user> --role <role> [--tenant <tenant>] 2014-06-12 09:44:56.123 | keystone user-role-add: error: argument --user/--user-id/--user_id: expected one argument 2014-06-12 09:44:56.164 | + get_state_from_hosts
could be networking I guess? But the overcloud controller log shows keystone being restarted after we start using it. Thats strange.
"""

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/104510

Changed in tripleo:
assignee: nobody → Derek Higgins (derekh)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/104602

Changed in tripleo:
assignee: Derek Higgins (derekh) → Clint Byrum (clint-fewbar)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/104602
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1c9c8249a2f245c053d6ee9ef088d49437a98fc1
Submitter: Jenkins
Branch: master

commit 1c9c8249a2f245c053d6ee9ef088d49437a98fc1
Author: Clint Byrum <email address hidden>
Date: Thu Jul 3 09:08:47 2014 -0700

    Make stack complete depend on all nodes

    Previously the completion signal was just based on the first run of
    os-refresh-config. But in this case, we actually need to wait until it
    runs successfully with all hosts computed. That way we can know that
    services aren't in an unstable state while that configuration rolls out.

    Change-Id: I3b965c19c92b366df3069cb8e1daffa18252c884
    Closes-Bug: #1337230

Changed in tripleo:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Derek Higgins (<email address hidden>) on branch: master
Review: https://review.openstack.org/104510

Jay Dobies (jdob)
Changed in tripleo:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.