AMQPConnectionException on launch VM after restart controller

Environment: nova stable/diablo in two machine cluster, one controller and one compute node.

Problem: After rebooting the controller and trying to launch a new VM (a few minutes after the controller is back up), the launched VM goes into an error state. If retry launching VM again it works.

Stack Trace from nova-compute.log file:

2012-01-03 17:14:18,624 AUDIT nova.compute.manager [6b505190-e161-466e-b6ad-4b61358fda20 nova project_503] instance 32: starting...
2012-01-03 17:14:18,680 DEBUG nova.rpc [6b505190-e161-466e-b6ad-4b61358fda20 nova project_503] Making asynchronous call on network ... from (pid=21128) multicall /usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:730
2012-01-03 17:14:18,680 DEBUG nova.rpc [6b505190-e161-466e-b6ad-4b61358fda20 nova project_503] MSG_ID is 1b804a7965464cbca5ab63e42760a0f6 from (pid=21128) multicall /usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:733
2012-01-03 17:14:18,681 ERROR nova.compute.manager [6b505190-e161-466e-b6ad-4b61358fda20 nova project_503] Instance '32' failed network setup.
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 411, in _logging_error
(nova.compute.manager): TRACE: yield
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 442, in _run_instance
(nova.compute.manager): TRACE: network_info = _make_network_info()
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 378, in _make_network_info(nova.compute.manager): TRACE: requested_networks=requested_networks)(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/network/api.py", line 162, in allocate_for_instance
(nova.compute.manager): TRACE: 'args': args})(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 45, in call
(nova.compute.manager): TRACE: return get_impl().call(context, topic, msg)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 746, in call
(nova.compute.manager): TRACE: rv = multicall(context, topic, msg)(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 738, in multicall(nova.compute.manager): TRACE: conn.declare_direct_consumer(msg_id, wait_msg)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 449, in declare_direct_consumer
(nova.compute.manager): TRACE: self.declare_consumer(DirectConsumer, topic, callback)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 394, in declare_consumer
(nova.compute.manager): TRACE: self.consumer_num.next())
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 140, in __init__
(nova.compute.manager): TRACE: **options)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 61, in __init__
(nova.compute.manager): TRACE: self.reconnect(channel)
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 68, in reconnect
(nova.compute.manager): TRACE: self.queue.declare()
(nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 359, in declare(nova.compute.manager): TRACE: return (self.name and self.exchange.declare(nowait),
(nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 151, in declare
(nova.compute.manager): TRACE: nowait=nowait)
(nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/syn.py", line 14, in blocking
(nova.compute.manager): TRACE: return __sync_current(fun, *args, **kwargs)
(nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/syn.py", line 40, in __eblocking__
(nova.compute.manager): TRACE: return spawn(fun, *args, **kwargs).wait()
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
(nova.compute.manager): TRACE: return self._exit_event.wait()
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait(nova.compute.manager): TRACE: return hubs.get_hub().switch()
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
(nova.compute.manager): TRACE: return self.greenlet.switch()
(nova.compute.manager): TRACE: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
(nova.compute.manager): TRACE: result = function(*args, **kwargs)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/channel.py", line 843, in exchange_declare
(nova.compute.manager): TRACE: (40, 11), # Channel.exchange_declare_ok
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/abstract_channel.py", line 89, in wait
(nova.compute.manager): TRACE: self.channel_id, allowed_methods)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/connection.py", line 218, in _wait_method
(nova.compute.manager): TRACE: self.wait()
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/abstract_channel.py", line 105, in wait
(nova.compute.manager): TRACE: return amqp_method(self, args)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/amqplib/client_0_8/connection.py", line 367, in _close
(nova.compute.manager): TRACE: raise AMQPConnectionException(reply_code, reply_text, (class_id, method_id))
(nova.compute.manager): TRACE: AMQPConnectionException: (320, u"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'", (0, 0), '')
(nova.compute.manager): TRACE: