Glance

[sru] 'unhashable type' when sending notifications via Qpid

Bug #980872 reported by Jiang Yong on 2012-04-13

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Glance	Fix Released	High	Russell Bryant	Glance 2012.2 "folsom"
Essex	Fix Released	Undecided	Unassigned	Glance 2012.1.1
glance (Ubuntu)	Fix Released	Undecided	Unassigned
Precise	Fix Released	Undecided	Chuck Short	Ubuntu ubuntu-12.04.1

Bug Description

When I set notifier_strategy = noop and add a image to glance it works well, but when I set notifier_strategy = qpid and add a image to glance it doesn't work! I have check my qpid with qpid's examples and it works well!

The error message is here:

[openstack@cc images]$ glance -I glance -K service123 -T service -N http://127.0.0.1:5000/v2.0 add name="tty-linux-kernel" disk_format=aki container_format=aki < ttylinux-uec-amd64-12.1_2.6.35-22_1-vmlinuz
Uploading image 'tty-linux-kernel'
=====================================================================================================================================================[100%] 13.9M/s, ETA 0h 0m 0s
Failed to add image. Got error:
The request returned 500 Internal Server Error

The response body:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/eventlet/wsgi.py", line 336, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/glance-2012.1-py2.6.egg/glance/common/wsgi.py", line 279, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.6/site-packages/webob/request.py", line 1053, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.6/site-packages/webob/request.py", line 1022, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.6/site-packages/keystone-2012.1-py2.6.egg/keystone/middleware/auth_token.py", line 176, in __call__
    return self.app(env, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/glance-2012.1-py2.6.egg/glance/common/wsgi.py", line 279, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.6/site-packages/webob/request.py", line 1053, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.6/site-packages/webob/request.py", line 1022, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/glance-2012.1-py2.6.egg/glance/common/wsgi.py", line 477, in __call__
    request, **action_args)
  File "/usr/lib/python2.6/site-packages/glance-2012.1-py2.6.egg/glance/common/wsgi.py", line 494, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/glance-2012.1-py2.6.egg/glance/api/v1/images.py", line 606, in create
    image_meta = self._handle_source(req, id, image_meta, image_data)
  File "/usr/lib/python2.6/site-packages/glance-2012.1-py2.6.egg/glance/api/v1/images.py", line 545, in _handle_source
    image_meta = self._upload_and_activate(req, image_meta)
  File "/usr/lib/python2.6/site-packages/glance-2012.1-py2.6.egg/glance/api/v1/images.py", line 536, in _upload_and_activate
    location = self._upload(req, image_meta)
  File "/usr/lib/python2.6/site-packages/glance-2012.1-py2.6.egg/glance/api/v1/images.py", line 465, in _upload
    self.notifier.error('image.upload', msg)
  File "/usr/lib/python2.6/site-packages/glance-2012.1-py2.6.egg/glance/notifier/__init__.py", line 73, in error
    self.strategy.error(msg)
  File "/usr/lib/python2.6/site-packages/glance-2012.1-py2.6.egg/glance/notifier/notify_qpid.py", line 150, in error
    self.sender_error.send(qpid_msg)
  File "<string>", line 6, in send
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 836, in send
    self._ecwait(lambda: self.linked)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
    result = self._ewait(lambda: self.closed or predicate(), timeout)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 787, in _ewait
    result = self.session._ewait(lambda: self.error or predicate(), timeout)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 554, in _ewait
    result = self.connection._ewait(lambda: self.error or predicate(), timeout)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 197, in _ewait
    self.check_error()
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 190, in check_error
    raise self.error
InternalError: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 507, in dispatch
    self.engine.dispatch()
  File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 811, in dispatch
    self.process(ssn)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 1046, in process
    self.send(snd, msg)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 1274, in send
    sst.write_cmd(xfr, msg_acked, sync=msg._sync)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 158, in write_cmd
    self.write_op(cmd)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 171, in write_op
    self.driver.write_op(op)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 684, in write_op
    self._seg_enc.write(*self._op_enc.read())
  File "/usr/lib/python2.6/site-packages/qpid/framing.py", line 178, in write
    payload = remaining[:self.max_payload]
TypeError: unhashable type

Note: Your image metadata may still be in the registry, but the image's status will likely be 'killed'.

Tags:

Related branches

lp:~openstack-ubuntu-testing/glance/precise-essex-proposed

Merged into lp:~ubuntu-server-dev/glance/essex at revision 123

Ubuntu Server Developers: Pending requested 2012-03-16

lp:ubuntu/precise-proposed/glance

Revision history for this message

Brian Waldon (bcwaldon) wrote on 2012-04-23:

Can you dig up the relevant logs? I'm curious what the message was that couldnt be sent.

Changed in glance:
status:	New → Incomplete

Brian Waldon (bcwaldon) on 2012-04-23

Changed in glance:
assignee:	nobody → Russell Bryant (russellb)
status:	Incomplete → In Progress
importance:	Undecided → High

Revision history for this message

Russell Bryant (russellb) wrote on 2012-04-23:

from IRC: http://paste.openstack.org/show/13490/

bug appears to be from trying to send a dict. it needs to get explicitly json encoded

Changed in glance:
status:	In Progress → Triaged
importance:	High → Medium

Revision history for this message

Brian Waldon (bcwaldon) wrote on 2012-04-23:

Looks like we can't set the qpid message contents to a dictionary. Russell is going to tackle this one.

Changed in glance:
milestone:	none → folsom-1

Russell Bryant (russellb) on 2012-04-23

Changed in glance:
importance:	Medium → High

Revision history for this message

Jiang Yong (jiangy) wrote on 2012-04-24:

My OS is CentOS6.2 and the qpid server and client are auto installed when I installed the OS. I only installed the keystone and glance with source code from git with the version essex. I hope these infomation are help for solving the problem.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-04-24: Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/6760

Changed in glance:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-04-24: Fix proposed to glance (stable/essex)

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/6761

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-04-24: Fix merged to glance (master)

Reviewed: https://review.openstack.org/6760
Committed: http://github.com/openstack/glance/commit/5bed23cbc962d3c6503f0ff93e6d1e326efbd49d
Submitter: Jenkins
Branch: master

commit 5bed23cbc962d3c6503f0ff93e6d1e326efbd49d
Author: Russell Bryant <email address hidden>
Date: Tue Apr 24 12:24:39 2012 -0400

Fix content type for qpid notifier.

Fix bug 980872.

    This patch fixes a regression I introduced in
    2d36facf14f4eb2742ba46274e04a73b5231aece. In that patch, I adjusted the
    content_type for messages sent with the qpid notifier to be
    'application/json' to match a change that went into the kombu notifier.
    Unfortunately, it's wrong.

    I assumed based on the kombu change that notifications were being json
    encoded before being passed into the notification driver. That's not
    the case. The message is a dict. So, just revert the change to set the
    content_type and let Qpid encode the notification as 'amqp/map'.

Change-Id: Iea027409f6200109b97fed93073cff84bc915536

Changed in glance:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-04-27: Fix merged to glance (stable/essex)

Reviewed: https://review.openstack.org/6761
Committed: http://github.com/openstack/glance/commit/5838b6390353e4c34762828684cee90410e4f8b1
Submitter: Jenkins
Branch: stable/essex

commit 5838b6390353e4c34762828684cee90410e4f8b1
Author: Russell Bryant <email address hidden>
Date: Tue Apr 24 12:24:39 2012 -0400

Fix content type for qpid notifier.

Fix bug 980872.

(cherry picked from commit 5bed23cbc962d3c6503f0ff93e6d1e326efbd49d)

Change-Id: I8ba039612d9603377028ba72cb80ae89d675c741

tags:

added: in-stable-essex

Thierry Carrez (ttx) on 2012-05-23

Changed in glance:
status:	Fix Committed → Fix Released

Revision history for this message

Chuck Short (zulcss) wrote on 2012-06-07: Re: 'unhashable type' when sending notifications via Qpid

** Impact **

A regression was introduced in essex, it was wrongly assumed that the content_type for messages sent with the qpid notifier to be 'application/json' to match a change that went into the kombu notifier.

** Developement Fix **

Fixed in the development trunk: https://review.openstack.org/6760 and in quantal

** Stable Fix **

Fixed in the stable/essex branch at: https://review.openstack.org/6761

** Test Case **

Run the glance.tests.unit.test_notifier

** Regression Potential **

None, this is not enabled by default in Ubuntu.

Chuck Short (zulcss) on 2012-06-08

summary:	- 'unhashable type' when sending notifications via Qpid + [sru] 'unhashable type' when sending notifications via Qpid
Changed in nova (Ubuntu Precise):
milestone:	none → ubuntu-12.04.1
assignee:	nobody → Chuck Short (zulcss)

Chuck Short (zulcss) on 2012-06-11

affects:	nova (Ubuntu) → glance (Ubuntu)
Changed in glance (Ubuntu):
status:	New → In Progress
Changed in glance (Ubuntu Precise):
status:	New → In Progress

Revision history for this message

Brian Murray (brian-murray) wrote on 2012-06-12: Please test proposed package

#10

Hello ugyn, or anyone else affected,

Accepted glance into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in glance (Ubuntu Precise):
status:	In Progress → Fix Committed
tags:	added: verification-needed

Revision history for this message

Adam Gandelman (gandelman-a) wrote on 2012-07-09: Verification report.

#11

Please find the attached Jenkins job results from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Glance has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/6760
Stable review: https://review.openstack.org/6761

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Revision history for this message

Adam Gandelman (gandelman-a) wrote on 2012-07-09:

#12

2012.1+stable~20120608-5462295-0ubuntu2.2.log Edit (72.4 KiB, text/plain)

Test coverage log.

Adam Gandelman (gandelman-a) on 2012-07-09

tags:

added: verification-done
removed: verification-needed

Revision history for this message

Chris Halse Rogers (raof) wrote on 2012-07-10:

#13

This looks ready to move to -updates. What is the status of this in Quantal? We need to be sure that it'll be addressed there so that upgraders don't suffer a regression.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2012-07-10:

#14

This bug was fixed in the package glance - 2012.1+stable~20120608-5462295-0ubuntu2.2

---------------
glance (2012.1+stable~20120608-5462295-0ubuntu2.2) precise-proposed; urgency=low

  * New usptream snapshot. (LP: #1010473)
  * Resynchronize with stable/essex:
   - 5462295 Fix i18n in glance.notifier.notify_kombu. (LP: #983829)
   - 7a9e3a7 Omit Content-Length on chunked transfer. (LP: #981332)
   - 5838b63 Fix content type for qpid notifier. (LP: #980872)
   - 98913da search for logger in PATH. Fixes (LP: #978907)
   - f136e7e Ensure swift auth URL includes trailing slash. Fixes (LP: #979745)
  * debian/rules: Add ability to build tarballs from a git snapshot.
  * debian/patches/fix-pep8-ubuntu.patch: Fix pep8 errors.
-- Chuck Short <email address hidden> Tue, 05 Jun 2012 10:43:12 -0400