OpenStack Compute (nova)

nova-api should check UTF8 char in parameters

Bug #940430 reported by Nachi Ueno on 2012-02-24

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Invalid	Medium	Unassigned

Bug Description

I got following error.

root@localhost:~# nova --debug list
connect: (keystone.thefreecloud.org, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: keystone.thefreecloud.org:5000\r\nContent-Length: 108\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: python-novaclient\r\n\r\n{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "XXXXXXXXX"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Content-Length: 1148
header: Date: Fri, 24 Feb 2012 16:13:00 GMT
connect: (nova-api.thefreecloud.org, 8774)
send: u'GET /v1.1/1/servers/detail HTTP/1.1\r\nHost: nova-api.thefreecloud.org:8774\r\nx-auth-project-id: admin\r\nx-auth-token: XXXXXXXXXXXXXXXXXX g\r\naccept-encoding: gzip, deflate\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 500 Internal Server Error\r\n'
header: Content-Length: 133
header: Content-Type: application/json; charset=UTF-8
header: Date: Fri, 24 Feb 2012 16:13:00 GMT
Traceback (most recent call last):
  File "/usr/local/bin/nova", line 9, in <module>
    load_entry_point('python-novaclient==2012.1', 'console_scripts', 'nova')()
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 338, in main
    OpenStackComputeShell().main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 289, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 480, in do_list
    utils.print_list(cs.servers.list(search_opts=search_opts), columns,
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 247, in list
    return self._list("/servers%s%s" % (detail, query_string), "servers")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 69, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 130, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 118, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 101, in request
    raise exceptions.from_response(resp, body)
novaclient.exceptions.ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

I got error

byte 0xe7 in position 8: unexpected end of data
(nova.api.openstack): TRACE: Traceback (most recent call last):
(nova.api.openstack): TRACE: File "/opt/stack/nova/nova/api/openstack/__init__.py", line 64, in __call__
(nova.api.openstack): TRACE: return req.get_response(self.application)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/webob/request.py", line 1053, in get_response
(nova.api.openstack): TRACE: application, catch_exc_info=False)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/webob/request.py", line 1022, in call_application
(nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 212, in __call__
(nova.api.openstack): TRACE: return self._forward_request(env, start_response, proxy_headers)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 344, in _forward_request
(nova.api.openstack): TRACE: return self.app(env, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in __call__
(nova.api.openstack): TRACE: response = self.app(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in __call__
(nova.api.openstack): TRACE: response = self.app(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/webob/dec.py", line 147, in __call__
(nova.api.openstack): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/webob/dec.py", line 208, in call_func
(nova.api.openstack): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack): TRACE: File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 508, in __call__
(nova.api.openstack): TRACE: action=action)
(nova.api.openstack): TRACE: File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 428, in serialize
(nova.api.openstack): TRACE: self.serialize_body(response, response_data, content_type, action)
(nova.api.openstack): TRACE: File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 438, in serialize_body
(nova.api.openstack): TRACE: response.body = serializer.serialize(data, action)
(nova.api.openstack): TRACE: File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 283, in serialize
(nova.api.openstack): TRACE: return self.dispatch(data, action=action)
(nova.api.openstack): TRACE: File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 70, in dispatch
(nova.api.openstack): TRACE: return action_method(*args, **kwargs)
(nova.api.openstack): TRACE: File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 293, in default
(nova.api.openstack): TRACE: return utils.dumps(data)
(nova.api.openstack): TRACE: File "/opt/stack/nova/nova/utils.py", line 604, in dumps
(nova.api.openstack): TRACE: return json.dumps(value)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
(nova.api.openstack): TRACE: return _default_encoder.encode(obj)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
(nova.api.openstack): TRACE: chunks = self.iterencode(o, _one_shot=True)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
(nova.api.openstack): TRACE: return _iterencode(o, 0)
(nova.api.openstack): TRACE: UnicodeDecodeError: 'utf8' codec can't decode byte 0xe7 in position 8: unexpected end of data
(nova.api.openstack): TRACE:

nova --debug image-list was OK

I could reproduce this error with curl.
curl -H "x-auth-token: XXXXXXXXXXXXXXXXXXXXXXX" https://nova-api.trystack.org:9774/v1.1/1/servers/detail --insecure
{"cloudServersFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

Then I checked instance name,
SELECT id,display_name FROM `instances`

Then I found an instance use uft08 char in display name.
I removed this instance, then nova list command back to normal.

nova-api should check UTF8 char in parameters

Tags:

Revision history for this message

Pádraig Brady (p-draigbrady) wrote on 2012-02-24:

By utf8 char, I think you mean non ASCII char.
In fact utf8 chars are handled by default:

>>> json.dumps(u"\xe7")
'"\\u00e7"'
>>> json.dumps(u"\xe7".encode("utf8"))
'"\\u00e7"'
>>> json.dumps("\xe7", encoding="iso8859-1")
'"\\u00e7"'
>>> json.dumps("\xe7")
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe7 in position 0: unexpected end of data

So well need to convert this string to utf8 or unicode
I'll take a look at where best to do this conversion

Changed in nova:
status:	New → In Progress

Revision history for this message

Thierry Carrez (ttx) wrote on 2012-06-07:

Padraig: if you're not working on this, remove assignee and set back status to 'Confirmed'

Changed in nova:
assignee:	nobody → Pádraig Brady (p-draigbrady)
importance:	Undecided → Medium

Revision history for this message

Thierry Carrez (ttx) wrote on 2012-12-13:

Looks like you're not working on that anymore. Please set back to InProgress and reassign to you if you're working on proposing a change for merging.

Changed in nova:
assignee:	Pádraig Brady (p-draigbrady) → nobody
status:	In Progress → Confirmed

Revision history for this message

Joshua Harlow (harlowja) wrote on 2013-03-29:

I think this might be in the API also.

https://bugs.launchpad.net/nova/+bug/1160487

Its not as easy to trigger there, but the API itself sends massive queries back and forth when a large number of instances exist under a given user, basically DOSing the nova-api process...

DongJin OH (ehwlstme) on 2013-07-22

Changed in nova:
assignee:	nobody → DongJin OH (ehwlstme)
status:	Confirmed → In Progress

DongJin OH (ehwlstme) on 2013-07-22

Changed in nova:
status:	In Progress → Confirmed
assignee:	DongJin OH (ehwlstme) → nobody
assignee:	nobody → DongJin OH (ehwlstme)
status:	Confirmed → In Progress

Tiantian Gao (gtt116) on 2013-07-22

tags:

added: api

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-07-30: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/39326

Revision history for this message

Brian Elliott (belliott) wrote on 2013-08-01:

I have seen similar errors before. I suspect the configuration of your database is the issue. Check the charset it's using.

AFAIK, UTF8 server names are permissible.

Revision history for this message

June Yi (gochist) wrote on 2013-08-02:

#10

In my devstack environment, it can be reproduced by booting an instance with name contains utf8 char and then listing instances while the charset of database is utf8.

Revision history for this message

DongJin OH (ehwlstme) wrote on 2013-08-02:

#11

If create instance by UTF8 type name, host name of instance is default empty value(not display_name). so I think that not being creation is better way, if the instance name is UTF8 type. Do you think that permitting the UTF8 is better than modyfing API?

Also, I think this error is not my database configuration problem. My database configuration is UTF8 type.

Revision history for this message

Brian Elliott (belliott) wrote on 2013-08-06:

#12

Okay, not sure why this is occurring then. I feel that removing support for UTF8 names is the wrong approach, though.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-08-15:

#13

Fix proposed to branch: master
Review: https://review.openstack.org/42054

Joe Gordon (jogo) on 2014-06-06

Changed in nova:
status:	In Progress → Invalid
status:	Invalid → Confirmed

Christopher Yeoh (cyeoh-0) on 2014-09-29

Changed in nova:
assignee:	DongJin OH (ehwlstme) → nobody

Revision history for this message

Christopher Yeoh (cyeoh-0) wrote on 2014-09-29:

#14

This doesn't appear to be reproducible anymore and we now handle utf8 host names correctly

Changed in nova:
status:	Confirmed → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.