OpenStack Identity (keystone)

keystone CLI shouldn't traceback when invalid data entered:UnboundLocalError: local variable 'metadata_ref' referenced before assignment

Bug #958135 reported by Kevin Jackson
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Critical
Chmouel Boudjnah
OpenStack Identity (keystone) 2012.1 "essex"

Bug Description

Per Chmouel's debugging run:

 - this occurred when "name" was entered into keystone, but "ID" (the UUID) was needed. The command line options are not terribly clear on what's needed.

-joe

ii keystone 2012.1~rc1~20120316.2145-0ubuntu1 OpenStack identity service - Daemons
ii python-keystone 2012.1~rc1~20120316.2145-0ubuntu1 OpenStack identity service - Python library
ii python-keystoneclient 2012.1~rc1~20120310.0-0ubuntu1 Client libary for Openstack Keystone API
ii nova-api 2012.1~rc1~20120316.13416-0ubuntu1 OpenStack Compute - API frontend
ii nova-cert 2012.1~rc1~20120316.13416-0ubuntu1 OpenStack Compute - certificate management
ii nova-common 2012.1~rc1~20120316.13416-0ubuntu1 OpenStack Compute - common files
ii nova-compute 2012.1~rc1~20120316.13416-0ubuntu1 OpenStack Compute - compute node
ii nova-compute-kvm 2012.1~rc1~20120316.13416-0ubuntu1 OpenStack Compute - compute node (KVM)
ii nova-network 2012.1~rc1~20120316.13416-0ubuntu1 OpenStack Compute - Network manager
ii nova-objectstore 2012.1~rc1~20120316.13416-0ubuntu1 OpenStack Compute - object store
ii nova-scheduler 2012.1~rc1~20120316.13416-0ubuntu1 OpenStack Compute - virtual machine scheduler
ii python-nova 2012.1~rc1~20120316.13416-0ubuntu1 OpenStack Compute Python libraries
ii python-novaclient 2012.1~rc1~20120310.525-0ubuntu1 client library for OpenStack Compute API

Trying to use euca2ools on Ubuntu I run into the follow stack trace from keystone:

(this was from euca-describe-instances)

/usr/lib/python2.7/dist-packages/webob/dec.py:142: DeprecationWarning: Response.request and Response.environ are deprecated
  req.response.request = req
/usr/lib/python2.7/dist-packages/webob/dec.py:142: DeprecationWarning: Response.request and Response.environ are deprecated
  req.response.request = req
/usr/lib/python2.7/dist-packages/webob/dec.py:142: DeprecationWarning: Response.request and Response.environ are deprecated
  req.response.request = req
/usr/lib/python2.7/dist-packages/webob/dec.py:142: DeprecationWarning: Response.request and Response.environ are deprecated
  req.response.request = req
/usr/lib/python2.7/dist-packages/webob/dec.py:142: DeprecationWarning: Response.request and Response.environ are deprecated
  req.response.request = req
(root): 2012-03-17 20:27:47,420 DEBUG wsgi __call__ ******************** REQUEST ENVIRON ********************
(root): 2012-03-17 20:27:47,420 DEBUG wsgi __call__ SCRIPT_NAME = /v2.0
(root): 2012-03-17 20:27:47,420 DEBUG wsgi __call__ webob.adhoc_attrs = {'response': <Response at 0x348b310 200 OK>}
(root): 2012-03-17 20:27:47,420 DEBUG wsgi __call__ REQUEST_METHOD = POST
(root): 2012-03-17 20:27:47,420 DEBUG wsgi __call__ PATH_INFO = /tokens
(root): 2012-03-17 20:27:47,420 DEBUG wsgi __call__ SERVER_PROTOCOL = HTTP/1.0
(root): 2012-03-17 20:27:47,420 DEBUG wsgi __call__ CONTENT_LENGTH = 434
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ eventlet.posthooks = []
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ SERVER_NAME = 172.16.0.1
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ REMOTE_ADDR = 172.16.0.1
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ eventlet.input = <eventlet.wsgi.Input object at 0x347ee50>
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ wsgi.url_scheme = http
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ webob._body_file = (<LimitedLengthFile(<eventlet.wsgi.Input object at 0x347ee50>, maxlen=434)>, <eventlet.wsgi.Input object at 0x347ee50>)
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ SERVER_PORT = 5000
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ wsgi.input = <_io.BytesIO object at 0x347fe90>
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ HTTP_HOST = 172.16.0.1:5000
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ wsgi.multithread = True
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ openstack.params = {u'auth': {u'OS-KSEC2:ec2Credentials': {u'access': u'9ff31cd92848406088fea3a2cc6f77f2', u'host': u'172.16.0.1:8773', u'verb': u'POST', u'params': {u'SignatureVersion': u'2', u'AWSAccessKeyId': u'9ff31cd92848406088fea3a2cc6f77f2', u'Timestamp': u'2012-03-17T20:26:30Z', u'SignatureMethod': u'HmacSHA256', u'Version': u'2010-08-31', u'Action': u'DescribeInstances'}, u'signature': u'2tyM3YNz9o0lOMdRmQtjl0ewR2qPeLahKvYdAvVAgqw=', u'path': u'/services/Cloud/'}}}
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ wsgi.version = (1, 0)
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ openstack.context = {'token_id': None, 'is_admin': False}
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ GATEWAY_INTERFACE = CGI/1.1
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ wsgi.run_once = False
(root): 2012-03-17 20:27:47,421 DEBUG wsgi __call__ wsgi.errors = <open file '<stderr>', mode 'w' at 0x7fe8a4c2a270>
(root): 2012-03-17 20:27:47,422 DEBUG wsgi __call__ wsgi.multiprocess = False
(root): 2012-03-17 20:27:47,422 DEBUG wsgi __call__ webob.is_body_seekable = True
(root): 2012-03-17 20:27:47,422 DEBUG wsgi __call__ CONTENT_TYPE = application/json
(root): 2012-03-17 20:27:47,422 DEBUG wsgi __call__ HTTP_ACCEPT_ENCODING = identity
(root): 2012-03-17 20:27:47,422 DEBUG wsgi __call__
(root): 2012-03-17 20:27:47,422 DEBUG wsgi __call__ ******************** REQUEST BODY ********************
(root): 2012-03-17 20:27:47,422 DEBUG wsgi __call__ {"auth": {"OS-KSEC2:ec2Credentials": {"access": "9ff31cd92848406088fea3a2cc6f77f2", "host": "172.16.0.1:8773", "verb": "POST", "params": {"SignatureVersion": "2", "AWSAccessKeyId": "9ff31cd92848406088fea3a2cc6f77f2", "Timestamp": "2012-03-17T20:26:30Z", "SignatureMethod": "HmacSHA256", "Version": "2010-08-31", "Action": "DescribeInstances"}, "signature": "2tyM3YNz9o0lOMdRmQtjl0ewR2qPeLahKvYdAvVAgqw=", "path": "/services/Cloud/"}}}
(root): 2012-03-17 20:27:47,422 DEBUG wsgi __call__
/usr/lib/python2.7/dist-packages/webob/dec.py:142: DeprecationWarning: Response.request and Response.environ are deprecated
  req.response.request = req
/usr/lib/python2.7/dist-packages/webob/dec.py:142: DeprecationWarning: Response.request and Response.environ are deprecated
  req.response.request = req
/usr/lib/python2.7/dist-packages/webob/dec.py:142: DeprecationWarning: Response.request and Response.environ are deprecated
  req.response.request = req
/usr/lib/python2.7/dist-packages/webob/dec.py:142: DeprecationWarning: Response.request and Response.environ are deprecated
  req.response.request = req
/usr/lib/python2.7/dist-packages/webob/dec.py:142: DeprecationWarning: Response.request and Response.environ are deprecated
  req.response.request = req
(root): 2012-03-17 20:27:47,425 DEBUG wsgi __call__ arg_dict: {}
(eventlet.wsgi.server): 2012-03-17 20:27:47,427 DEBUG wsgi write Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 336, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 293, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 293, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 293, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 293, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 316, in __call__
    resp = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 181, in __call__
    result = method(context, **params)
  File "/usr/lib/python2.7/dist-packages/keystone/service.py", line 350, in authenticate
    for role_id in metadata_ref.get('roles', []):
UnboundLocalError: local variable 'metadata_ref' referenced before assignment

(eventlet.wsgi.server): 2012-03-17 20:27:47,429 DEBUG wsgi write 172.16.0.1 - - [17/Mar/2012 20:27:47] "POST /v2.0/tokens HTTP/1.1" 500 5137 0.010099

See original description
Revision history for this message
Tom Ellis (tellis) wrote :
Download full text (4.9 KiB)

I'm having a similar issue with the ec2 commands, although my traceback is very different:
(root): 2012-03-18 22:20:28,196 DEBUG utils _calc_signature_2 base64 encoded digest: 5hkz+Ok1N4fy/6lBMvYu0e4A/Li3AYeGC07F3oj9RH8=
(eventlet.wsgi.server): 2012-03-18 22:20:28,199 DEBUG wsgi write Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 336, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 293, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 293, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 293, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 293, in __call__
    response = request.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in...

Read more...

Thierry Carrez (ttx)
tags: added: essex-rc-potential
Revision history for this message
Joseph Heck (heckj) wrote :

Kevin,

Can you provide a little more detail for me to reproduce this issue? What did you initialize your keystone environment with? And what back-end drivers are you using for Token, Identity, and Catalog?

Revision history for this message
Joseph Heck (heckj) wrote :

Tom - your POST is to a different API endpoint, so I think you've got a valuable bug in it's own right that should be separate from this one. Would you mind taking that detail and opening in another bug against keystone?

Please also add in the data so we can repro the issue - what backend drivers are you using for Token, Identity, and Catalog, and what have you initialized your environment with (users, tenants, roles, etc)?

Changed in keystone:
status: New → Incomplete
importance: Undecided → High
Revision history for this message
Kevin Jackson (kevin-linuxservices) wrote :

Hi Joseph - I use the following

EC2_PUBLIC_URL="http://$ENDPOINT:8773/services/Cloud"
EC2_ADMIN_URL="http://$ENDPOINT:8773/services/Admin"
EC2_INTERNAL_URL=$EC2_PUBLIC_URL

This I was unsure of so could explain things:

I've tried:
keystone --token $SERVICE_TOKEN --endpoint $SERVICE_ENDPOINT service-create --name ec2 --type ec2 --description 'EC2 Service'
and
keystone --token $SERVICE_TOKEN --endpoint $SERVICE_ENDPOINT service-create --name ec2 --type compute --description 'EC2 Service'

(Both produce same err when using euca2ools)

Then I get the relevant IDs and produce the endpoints
keystone --token $SERVICE_TOKEN --endpoint $SERVICE_ENDPOINT endpoint-create --region nova --service_id $ID --publicurl $PUBLIC --adminurl $ADMIN --internalurl $INTERNAL

Joseph Heck (heckj)
Changed in keystone:
status: Incomplete → Triaged
milestone: none → essex-rc1
importance: High → Critical
assignee: nobody → Chmouel Boudjnah (chmouel)
Revision history for this message
Chmouel Boudjnah (chmouel) wrote :

hi Kevin,

I would need some more information to help me reproduce.

can you please ping me tomorrow morning (22/03 GMT) so we can do some debug session over IRC ?

Thanks a lot,
Chmouel.

Revision history for this message
Dolph Mathews (dolph) wrote :

Any progress on this?

Revision history for this message
Chmouel Boudjnah (chmouel) wrote :

After a full debugging session, it seems that the problem was because kevin and Tom had user_name instead of user_id for the user_id field. This seems to come from the keystone command line as when creating new ec2 credentials --user is more likely to be a username when this should be a user-id.

I would vote to rename this to --user-id for RC2 as this can be a real pain (more a pain than renaming the docs, fix devstac etc..)

Revision history for this message
Joseph Heck (heckj) wrote :

Then this bug should transform to be "don't traceback when invalid data is entered" - but you're right on Chmouel with naming the arguments more appropriate in the CLI to what they need.

summary: - UnboundLocalError: local variable 'metadata_ref' referenced before
- assignment
+ keystone CLI shouldn't traceback when invalid data
+ entered:UnboundLocalError: local variable 'metadata_ref' referenced
+ before assignment
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/5713

Changed in keystone:
status: Triaged → In Progress
Revision history for this message
Kevin Jackson (kevin-linuxservices) wrote :

I have an update as I now have a working set up:

First problem:
my /etc/nova/nova.conf line was wrong initially

I had
    --keystone_ec2_url=http://172.29.0.1:5000/v2.0/tokens
instead of
    --keystone_ec2_url=http://172.29.0.1:5000/v2.0/ec2tokens

When I corrected this, this moved from the metadata_ref stack trace [#0], to the TypeError: 'NoneType' object has no attribute '__getitem__' (#1) stack trace. I assumed, same/similar bugs causing the issue.

At this point chmouel and I went on to debug the setup.
We found that I was using a named of my tenant, instead of the ID in my EC2 credentials. (#7)

With this in place, we still were getting the stack trace seen in #1.

On looking at this further - I then took the complete keystone.conf from source, including the local template catalog file and THIS WORKED.

Not to be deterred from using a MySQL backend for my catalog and I got this working too. Then looking through the diffs in the keystone.conf that is shipped with ubuntu's packages and the one you can find in Git shows these interesting lines

< ubuntu confs
> Git confs, which work

< driver = keystone.token.backends.sql.Token
---
> driver = keystone.token.backends.kvs.Token
 52c55
< driver = keystone.contrib.ec2.backends.sql.Ec2
---
> driver = keystone.contrib.ec2.backends.kvs.Ec2

Thierry Carrez (ttx)
tags: removed: essex-rc-potential
Revision history for this message
Dolph Mathews (dolph) wrote :

This is part of bug 963056, but I'd rather not mark as dupe.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/5713
Committed: http://github.com/openstack/keystone/commit/1904228a5a3fef549c5b9294eba5c39f9f6f72bd
Submitter: Jenkins
Branch: master

commit 1904228a5a3fef549c5b9294eba5c39f9f6f72bd
Author: Chmouel Boudjnah <email address hidden>
Date: Thu Mar 22 21:34:39 2012 +0000

    Check values for EC2.

    - Add multiple check to methods to make sure we have a proper
      tenant_id/user_id/credentials.
    - Fixes bug 958135.

    Change-Id: I4dd171e3db32d6ebdc70bb1a83492c8ecd09c21c

Changed in keystone:
status: In Progress → Fix Committed
Revision history for this message
Chmouel Boudjnah (chmouel) wrote :

This commit would not allow that kind of bug anymore. I think their is another bug somewhere where this kind of stuff should handled more gracefully but definitely not rc-critical. We need to follow-up as well why sql token are not working for Kevin.

Thierry Carrez (ttx)
Changed in keystone:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in keystone:
milestone: essex-rc1 → 2012.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.