PKI tokens are broken after 24 hours

Bug #1074172 reported by Vish Ishaya
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
High
Vish Ishaya
Folsom
Fix Released
High
Vish Ishaya

Bug Description

After 24 hours of being up the auth_token_middleware attempts to retrieve the certificate_revocation_list but fails because the admin token is expired. There is no retry logic in this code path like there is with the uuid tokens to generate a new admin token.

Revision history for this message
Vish Ishaya (vishvananda) wrote :

Quick untested fix

diff --git a/keystone/middleware/auth_token.py b/keystone/middleware/auth_token.py
index 5c198e8..8032b12 100644
--- a/keystone/middleware/auth_token.py
+++ b/keystone/middleware/auth_token.py
@@ -771,10 +771,16 @@ class AuthProtocol(object):
         with open(self.revoked_file_name, 'w') as f:
             f.write(value)

- def fetch_revocation_list(self):
+ def fetch_revocation_list(self, retry=True):
         headers = {'X-Auth-Token': self.get_admin_token()}
         response, data = self._json_request('GET', '/v2.0/tokens/revoked',
                                             additional_headers=headers)
+ if response.status == 401:
+ if retry:
+ LOG.info('Keystone rejected admin token %s, resetting',
+ headers)
+ self.admin_token = None
+ return self.fetch_revocation_list(False)
         if response.status != 200:
             raise ServiceError('Unable to fetch token revocation list.')
         if (not 'signed' in data):

Changed in keystone:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Vish Ishaya (vishvananda) wrote :

patch with proper whitespace

Joseph Heck (heckj)
Changed in keystone:
assignee: nobody → Adam Young (ayoung)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/15242

Changed in keystone:
assignee: Adam Young (ayoung) → Joseph Heck (heckj)
status: Triaged → In Progress
Revision history for this message
Adam Young (ayoung) wrote :

Each PKI token has the expiry in it. We should check that before any calls, and, if needs be, request a new token instead

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/15252

Joseph Heck (heckj)
Changed in keystone:
milestone: none → grizzly-1
Changed in keystone:
assignee: Joseph Heck (heckj) → Vish Ishaya (vishvananda)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/15242
Committed: http://github.com/openstack/keystone/commit/7cc02c80cfb1976271fa8b6271091fcd35c1cb34
Submitter: Jenkins
Branch: master

commit 7cc02c80cfb1976271fa8b6271091fcd35c1cb34
Author: Joe Heck <email address hidden>
Date: Thu Nov 1 15:36:31 2012 -0700

    fixes bug 1074172

    updated diablo token based on output from diablo/stable keystone
    added expiry to example tokens for test_auth_middleware
    added a stack based HTTP response to test_auth_middleware to verify
    sequencing

    Change-Id: I738b0e9c1a0e62ad86adb95ec0b73f621513f7d4

Changed in keystone:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in keystone:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/23334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/23468

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/folsom)

Reviewed: https://review.openstack.org/23468
Committed: http://github.com/openstack/keystone/commit/790c87e8bc401ef202e715a324a6053aaa9e2d5e
Submitter: Jenkins
Branch: stable/folsom

commit 790c87e8bc401ef202e715a324a6053aaa9e2d5e
Author: Vishvananda Ishaya <email address hidden>
Date: Mon Mar 4 13:37:46 2013 -0800

    Sync timeutils to pick up normalize fix.

    Required to backport fix for bug 1074172.

    Change-Id: I6003abedcfc6ba9d287cabda35c0bbc821519008

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/23334
Committed: http://github.com/openstack/keystone/commit/86901664189c62fce6f8f81619da0896cce469a1
Submitter: Jenkins
Branch: stable/folsom

commit 86901664189c62fce6f8f81619da0896cce469a1
Author: Joe Heck <email address hidden>
Date: Thu Nov 1 15:36:31 2012 -0700

    Backport of fix for 24-hour failure of pki.

    A 401 from the admin token was being treated as an invalid user token
    and the admin token was never re-requested. This would cause all
    services to incorrectly report all tokens as invalid after the service
    had been running for 24 hours. This change re-requests the admin token
    and attempts to revalidate instead of returning 401.

    Original commit message follows:

    fixes bug 1074172

    updated diablo token based on output from diablo/stable keystone
    added expiry to example tokens for test_auth_middleware
    added a stack based HTTP response to test_auth_middleware to verify
    sequencing

    Change-Id: I738b0e9c1a0e62ad86adb95ec0b73f621513f7d4
    (cherry picked from commit 7cc02c80cfb1976271fa8b6271091fcd35c1cb34)

Thierry Carrez (ttx)
Changed in keystone:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.