Glance

durability of rabbit exchange/queue should be configurable

Bug #1054183 reported by Eoghan Glynn
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
High
Eoghan Glynn
Glance 2012.2 "folsom"

Bug Description

The durability of the rabbit exchange and queue is hard-coded to True.

Yet for other openstack services using the common RPC code, this is configurable and defaults to False.

If there's a mismatch between the consumer and producer on the durable flag, this leads to a failure like:

  AMQPChannelException: (406, "PRECONDITION_FAILED - cannot redeclare exchange 'glance' in vhost with different type, durable, internal or autodelete value, (40, 10), 'Channel.exchange_declare'")

The consumption of glance notifications by ceilometer would be less fragile if glance durability could be similarly configured.

Eoghan Glynn (eglynn)
Changed in glance:
assignee: nobody → Eoghan Glynn (eglynn)
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/13474

Changed in glance:
status: New → In Progress
Eoghan Glynn (eglynn)
Changed in glance:
milestone: none → folsom-rc2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/13475

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/13474
Committed: http://github.com/openstack/glance/commit/eafed25ba64f5fd8af97bcbe8e9f12a1fe4f7c9f
Submitter: Jenkins
Branch: master

commit eafed25ba64f5fd8af97bcbe8e9f12a1fe4f7c9f
Author: Eoghan Glynn <email address hidden>
Date: Fri Sep 21 15:32:49 2012 +0000

    Add rabbit_durable_queues config option.

    Fixes bug 1054183

    Avoid AMQPChannelException: (406, "PRECONDITION_FAILED...") failures
    due to a mismatch between the durability of rabbitmq exchange/queue
    declared by glance and ceilometer.

    Change-Id: I4e25986a1f503782e701aa1168c4eb231ff25d06

Changed in glance:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (milestone-proposed)

Reviewed: https://review.openstack.org/13475
Committed: http://github.com/openstack/glance/commit/5268d6b1a71d1132a05cf989c322f44ec4445fb5
Submitter: Jenkins
Branch: milestone-proposed

commit 5268d6b1a71d1132a05cf989c322f44ec4445fb5
Author: Eoghan Glynn <email address hidden>
Date: Fri Sep 21 15:32:49 2012 +0000

    Add rabbit_durable_queues config option.

    Fixes bug 1054183

    Avoid AMQPChannelException: (406, "PRECONDITION_FAILED...") failures
    due to a mismatch between the durability of rabbitmq exchange/queue
    declared by glance and ceilometer.

    Change-Id: I4e25986a1f503782e701aa1168c4eb231ff25d06

Changed in glance:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in glance:
milestone: folsom-rc2 → 2012.2
Revision history for this message
Jay Pipes (jaypipes) wrote :

This has not been fixed...

On a fresh checkout of devstack, I now run into a glance service that won't respond. This, however, isn't the worst part. The worst part is that the glance API server is hung in a perpetual loop, eating up disk space for logs at an alarming rate.

After just a few seconds, the glance API log file (screen-ga-api.log) was at 17MB:

jpipes@uberbox:~/repos/devstack$ ls -alh /opt/stack/logs/screen-g-api*
-rw-rw-r-- 1 jpipes jpipes 17M Nov 1 13:56 /opt/stack/logs/screen-g-api.2012-11-01-134816.log
lrwxrwxrwx 1 jpipes jpipes 50 Nov 1 13:49 /opt/stack/logs/screen-g-api.log -> /opt/stack/logs/screen-g-api.2012-11-01-134816.log

The log file looks like this: http://paste.openstack.org/show/23512/

with the child process attempting to create the consumer, failing, and then the parent process launching another child process, which does the same thing, continuing ad-infinitum.

Basically, at a minimum, I believe that the AMQP negotiation should at least be *tried* before the child processes are spawned so that basic things like this can be caught before the child process starts. Either that, or the parent must have a wait to stop spawning on repeated failures.

Revision history for this message
Brian Waldon (bcwaldon) wrote :

We can definitely reproduce the mismatch of durability between the server and the client, but this bug covers the ability to configure the durability itself. I filed a new bug to track the issues (which I can replicate on my machine) that you're running into, Jay : https://bugs.launchpad.net/glance/+bug/1074132

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.