OpenStack Identity (keystone)

Deleting the 'admin' project in dashboard leads to TypeError: argument of type 'NoneType' is not iterable

Bug #1000609 reported by Jaroslav Henner
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Invalid
High
Unassigned

Bug Description

After deleting the admin project (created by openstack-keystone-sample-data) in dashboard, I got logged-out with this on login page:

Success: Deleted Project: admin
Error: Unable to retrieve project list.

keystone tenant-create --name admin
leads to following in keystone.log.

2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] wsgi.version = (1, 0)
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] openstack.context = {'token_id': None, 'is_admin': False}
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] GATEWAY_INTERFACE = CGI/1.1
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] wsgi.run_once = False
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] wsgi.errors = <open file '<stderr>', mode 'w' at 0x7f80ce3531e0>
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] wsgi.multiprocess = False
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] webob.is_body_seekable = True
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] CONTENT_TYPE = application/json
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] HTTP_ACCEPT_ENCODING = gzip, deflate
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi]
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ********************
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] {"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "****"}}}
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi]
2012-05-16 20:45:50 DEBUG [routes.middleware] Matched POST /tokens
2012-05-16 20:45:50 DEBUG [routes.middleware] Route path: '{path_info:.*}', defaults: {'controller': <keystone.service.PublicRouter object at 0x33170d0>}
2012-05-16 20:45:50 DEBUG [routes.middleware] Match dict: {'controller': <keystone.service.PublicRouter object at 0x33170d0>, 'path_info': '/tokens'}
2012-05-16 20:45:50 DEBUG [routes.middleware] Matched POST /tokens
2012-05-16 20:45:50 DEBUG [routes.middleware] Route path: '/tokens', defaults: {'action': u'authenticate', 'controller': <keystone.service.TokenController object at 0x3317490>}
2012-05-16 20:45:50 DEBUG [routes.middleware] Match dict: {'action': u'authenticate', 'controller': <keystone.service.TokenController object at 0x3317490>}
2012-05-16 20:45:50 DEBUG [keystone.common.wsgi] arg_dict: {}
2012-05-16 20:45:51 DEBUG [root] TOKEN_REF {'id': '8e7f42dc0a654165b8005d70466346f4', 'expires': datetime.datetime(2012, 5, 17, 18, 45, 51, 1282), 'user': {u'tenantId': None, u'enabled': True, u'email': <email address hidden>', 'name': u'admin', 'id': u'1a964e13a342430493057c2bc1912f35'}, 'tenant': None, 'metadata': {}}
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ********************
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] Content-Type = application/json
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] Vary = X-Auth-Token
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] Content-Length = 244
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi]
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ********************
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] {"access": {"token": {"expires": "2012-05-17T18:45:51Z", "id": "8e7f42dc0a654165b8005d70466346f4"}, "serviceCatalog": {}, "user": {"username": "admin", "roles_links": [], "id": "1a964e13a342430493057c2bc1912f35", "roles": [], "name": "admin"}}}
2012-05-16 20:45:51 DEBUG [eventlet.wsgi.server] 10.34.63.222 - - [16/May/2012 20:45:51] "POST /v2.0/tokens HTTP/1.1" 200 373 0.040573

2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] ******************** REQUEST ENVIRON ********************
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] SCRIPT_NAME = /v2.0
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] webob.adhoc_attrs = {'response': <Response at 0x39aeed0 200 OK>}
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] REQUEST_METHOD = POST
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] PATH_INFO = /tenants
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] SERVER_PROTOCOL = HTTP/1.0
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] CONTENT_LENGTH = 67
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] HTTP_X_AUTH_TOKEN = 8e7f42dc0a654165b8005d70466346f4
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] HTTP_USER_AGENT = python-keystoneclient
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] eventlet.posthooks = []
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] SERVER_NAME = 10.34.63.222
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] REMOTE_ADDR = 10.34.63.222
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] eventlet.input = <eventlet.wsgi.Input object at 0x39b3f50>
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] wsgi.url_scheme = http
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] SERVER_PORT = 5000
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] wsgi.input = <cStringIO.StringI object at 0x3a01c00>
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] HTTP_HOST = dell-r210ii-03.rhev.lab.eng.brq.redhat.com:5000
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] wsgi.multithread = True
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] openstack.params = {u'tenant': {u'enabled': True, u'name': u'demo3', u'description': None}}
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] wsgi.version = (1, 0)
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] openstack.context = {'token_id': '8e7f42dc0a654165b8005d70466346f4', 'is_admin': False}
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] GATEWAY_INTERFACE = CGI/1.1
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] wsgi.run_once = False
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] wsgi.errors = <open file '<stderr>', mode 'w' at 0x7f80ce3531e0>
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] wsgi.multiprocess = False
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] webob.is_body_seekable = True
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] CONTENT_TYPE = application/json
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] HTTP_ACCEPT_ENCODING = gzip, deflate
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi]
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ********************
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] {"tenant": {"enabled": true, "name": "demo3", "description": null}}
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi]
2012-05-16 20:45:51 DEBUG [routes.middleware] Matched POST /tenants
2012-05-16 20:45:51 DEBUG [routes.middleware] Route path: '{path_info:.*}', defaults: {'controller': <keystone.service.PublicRouter object at 0x33170d0>}
2012-05-16 20:45:51 DEBUG [routes.middleware] Match dict: {'controller': <keystone.service.PublicRouter object at 0x33170d0>, 'path_info': '/tenants'}
2012-05-16 20:45:51 DEBUG [routes.middleware] Matched POST /tenants
2012-05-16 20:45:51 DEBUG [routes.middleware] Route path: '/tenants', defaults: {'action': u'get_tenants_for_token', 'controller': <keystone.identity.core.TenantController object at 0x3317b10>}
2012-05-16 20:45:51 DEBUG [routes.middleware] Match dict: {'action': u'get_tenants_for_token', 'controller': <keystone.identity.core.TenantController object at 0x3317b10>}
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] arg_dict: {}
2012-05-16 20:45:51 ERROR [root] argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 184, in __call__
    result = method(context, **params)
  File "/usr/lib/python2.6/site-packages/keystone/identity/core.py", line 281, in get_tenants_for_token
    return self._format_tenant_list(tenant_refs, **params)
  File "/usr/lib/python2.6/site-packages/keystone/identity/core.py", line 355, in _format_tenant_list
    if 'enabled' not in x:
TypeError: argument of type 'NoneType' is not iterable
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ********************
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] Content-Type = application/json
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] Vary = X-Auth-Token
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] Content-Length = 187
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi]
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ********************
2012-05-16 20:45:51 DEBUG [keystone.common.wsgi] {"error": {"message": "An unexpected error prevented the server from fulfilling your request. argument of type 'NoneType' is not iterable", "code": 500, "title": "Internal Server Error"}}
2012-05-16 20:45:51 DEBUG [eventlet.wsgi.server] 10.34.63.222 - - [16/May/2012 20:45:51] "POST /v2.0/tenants HTTP/1.1" 500 335 0.015403

Joseph Heck (heckj)
Changed in keystone:
status: New → Triaged
importance: Undecided → Medium
importance: Medium → High
Revision history for this message
Robert Collins (lifeless) wrote :

get_tenants_for_token is passing None as one of the refs:
        for tenant_id in tenant_ids:
            tenant_refs.append(self.identity_api.get_tenant(
                context=context,
                tenant_id=tenant_id))
seems to be the proximate cause.

That said, your log trace shows the output from "keystone tenant-create --name demo3", not from" --name admin" AFAICT. Could you clarify please?

Revision history for this message
Robert Collins (lifeless) wrote :

The pam get_tenant will return None if user_ref['id'] were to be None.
The sql get_tenant does : [x.tenant_id for x in membership_refs] which will return None if the tenant_id is None, and nullable=False is not passed to the Column, so AFAICT that is the case. Possibly the membership ref is being update to NULL explicitly. Either way, the code and the schema are mismatched today: the code assumes the row can never be NULL, the schema does not prevent this.
The kvs store returns whatever is listed under tenants, I'd expect it to be much more likely to return stale tenants than None.
The ldap core looks reasonably immune.

Best bet on the data we have so far is the SQL one. Can you confirm you're using the SQL backend ?

Revision history for this message
Robert Collins (lifeless) wrote :

Checked the sqlalchemy code, the deleted state is a dict, so the carefully ordered code in the sql delete_tenant and delete_user is not guaranteed.

commit 3c9c38a8e08dd0300a04edb843a0b3e49486e86f
Merge: 4c8a101 2e912f4
Author: Jenkins <email address hidden>
Date: Tue May 15 07:05:04 2012 +0000

    Merge "Flush tenant membership deletion before user"

2 days prior to this bug being reported, would probably fix the symptoms (but not the underlying cause) - its still racy today. See the referenced bug 998137. A small tweak will make this more obvious I think, as an initial refactoring step before changing the schema.

Revision history for this message
Robert Collins (lifeless) wrote :
Download full text (3.3 KiB)

I'll put this in gerrit once my application for the -cla team is approved, but to start with:

This just makes the code cleaner; I need to track down whether openstack has backfill helpers or just assumes downtime is ok for the change to have the join table non-nullable.

diff --git a/keystone/identity/backends/sql.py b/keystone/identity/backends/sql.py
index dafd19b..d13683c 100644
--- a/keystone/identity/backends/sql.py
+++ b/keystone/identity/backends/sql.py
@@ -385,27 +385,13 @@ class Identity(sql.Base, identity.Driver):

     def delete_user(self, user_id):
         session = self.get_session()
- user_ref = session.query(User).filter_by(id=user_id).first()
- if not user_ref:
- raise exception.UserNotFound(user_id=user_id)
- membership_refs = session.query(UserTenantMembership)\
- .filter_by(user_id=user_id)\
- .all()
- metadata_refs = session.query(Metadata)\
- .filter_by(user_id=user_id)\
- .all()
-
         with session.begin():
- if membership_refs:
- for membership_ref in membership_refs:
- session.delete(membership_ref)
- session.flush()
- if metadata_refs:
- for metadata_ref in metadata_refs:
- session.delete(metadata_ref)
-
- session.delete(user_ref)
- session.flush()
+ session.query(UserTenantMembership)\
+ .filter_by(user_id=user_id).delete(False)
+ session.query(Metadata)\
+ .filter_by(user_id=user_id).delete(False)
+ if not session.query(User).filter_by(id=user_id).delete(False):
+ raise exception.UserNotFound(user_id=user_id)

     @handle_conflicts(type='tenant')
     def create_tenant(self, tenant_id, tenant):
@@ -438,26 +424,13 @@ class Identity(sql.Base, identity.Driver):

     def delete_tenant(self, tenant_id):
         session = self.get_session()
- tenant_ref = session.query(Tenant).filter_by(id=tenant_id).first()
- if not tenant_ref:
- raise exception.TenantNotFound(tenant_id=tenant_id)
- membership_refs = session.query(UserTenantMembership)\
- .filter_by(tenant_id=tenant_id)\
- .all()
- metadata_refs = session.query(Metadata)\
- .filter_by(tenant_id=tenant_id)\
- .all()
-
         with session.begin():
- if membership_refs:
- for membership_ref in membership_refs:
- session.delete(membership_ref)
- if metadata_refs:
- for metadata_ref in metadata_refs:
- session.delete(metadata_ref)
-
- session.delete(tenant_ref)
- session.flush()
+ session.query(UserTenantMembership)\
+ .filter_by(tenant_id=tenant_id).delete(False)
+ session.query(Metadata)\
+ .filter_by(tenant_id=tenant_id).delete(False)
+ if not sessio...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/11237

Changed in keystone:
status: Triaged → In Progress
Revision history for this message
Robert Collins (lifeless) wrote :

https://review.openstack.org/#/c/11237/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/11237
Committed: http://github.com/openstack/keystone/commit/37ff759707f64ee4f4eafee8448ece36b96173e4
Submitter: Jenkins
Branch: master

commit 37ff759707f64ee4f4eafee8448ece36b96173e4
Author: Robert Collins <email address hidden>
Date: Sun Aug 12 19:30:35 2012 +1200

    Simplify the sql backend deletion of users and tenants.

    There is a remaining problem in that the table definition permits dangling
    membership in tenants, and vice verca, but this change will make the fix for
    that easier to review, and make the code simpler and faster at the sametime.

    See bug 1000609 for the bug report that lead to examining this.

    Change-Id: Id7cd5fad7032779d352a7c577c8d10558091d767

Changed in keystone:
status: In Progress → Fix Committed
Robert Collins (lifeless)
Changed in keystone:
assignee: nobody → Robert Collins (lifeless)
assignee: Robert Collins (lifeless) → nobody
Thierry Carrez (ttx)
Changed in keystone:
milestone: none → folsom-3
status: Fix Committed → Fix Released
Revision history for this message
Thierry Carrez (ttx) wrote :

Fix is unrelated to bug.

Changed in keystone:
milestone: folsom-3 → none
status: Fix Released → Confirmed
Revision history for this message
Dolph Mathews (dolph) wrote :

I don't see how this is reproducible in the current codebase; calling get_tenants_for_token in this case would now result in a 404, and the code cited in the back trace is never reached. Please re-open if I'm mistaken.

Changed in keystone:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.