OK. I just did it again. I am not sure what "git commit tip" means. The running out of addresses was a one-time issue I had before understanding that the leased IP addresses for the vms are not returned immediately when the vms are terminated.
I am assuming that this kind of error and stacktrace in the log would be considered a bug even if it does not cause any obvious misbehavior immediately.
I used the current version of devstack but with a modified stack.sh from last week. I modified it to log to files in /var/log/nova instead of stdout so you have to create that directory before running. I used one compute node running api,network,compute and one controller node running everything. After running the modified stack.sh I exported the shell variables in exercise.sh, modified the min count in nova client to create 10 vms, and did
nova boot --flavor 1 --image 3 testvms
In this case the vms seen to have started but there is the AMQP error stacktrace in the log. When this happens, sometimes all the vms come up and sometimes not. I have attached:
1. nova-network log with stacktrace
2. modified stack.sh
3. localrc for both controller and compute
4. nova.conf (even though it is generated)
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 628, in _process_data
(nova.rpc): TRACE: ctxt.reply(None, None)
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 673, in reply
(nova.rpc): TRACE: msg_reply(self.msg_id, *args, **kwargs)
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 781, in msg_reply
(nova.rpc): TRACE: conn.direct_send(msg_id, msg)
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 562, in __exit__
(nova.rpc): TRACE: self._done()
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 547, in _done
(nova.rpc): TRACE: self.connection.reset()
(nova.rpc): TRACE: File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 382, in reset
(nova.rpc): TRACE: self.channel.close()
(nova.rpc): TRACE: File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 201, in close
(nova.rpc): TRACE: super(Channel, self).close()
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 194, in close
(nova.rpc): TRACE: (20, 41), # Channel.close_ok
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
(nova.rpc): TRACE: return self.dispatch_method(method_sig, args, content)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method
(nova.rpc): TRACE: return amqp_method(self, args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close
(nova.rpc): TRACE: (class_id, method_id))
(nova.rpc): TRACE: AMQPChannelException: (404, u"NOT_FOUND - no exchange '5b9a19d1b5e849d7ba749fada648a375' in vhost '/'", (60, 40), 'Channel.basic_publish')
(nova.rpc): TRACE:
2011-10-20 10:45:24,724 ERROR nova.rpc [-] Returning exception (404, u"NOT_FOUND - no exchange '5b9a19d1b5e849d7ba749fada648a375' in vhost '/'", (60, 40), 'Channel.basic_publish') to caller
2011-10-20 10:45:24,724 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 628, in _process_data\n ctxt.reply(None, None)\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 673, in reply\n msg_reply(self.msg_id, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 781, in msg_reply\n conn.direct_send(msg_id, msg)\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 562, in __exit__\n self._done()\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 547, in _done\n self.connection.reset()\n', ' File "/opt/stack/nova/nova/rpc/impl_kombu.py", line 382, in reset\n self.channel.close()\n', ' File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 201, in close\n super(Channel, self).close()\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 194, in close\n (20, 41), # Channel.close_ok\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait\n return self.dispatch_method(method_sig, args, content)\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method\n return amqp_method(self, args)\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close\n (class_id, method_id))\n', 'AMQPChannelException: (404, u"NOT_FOUND - no exchange \'5b9a19d1b5e849d7ba749fada648a375\' in vhost \'/\'", (60, 40), \'Channel.basic_publish\')\n']
OK. I just did it again. I am not sure what "git commit tip" means. The running out of addresses was a one-time issue I had before understanding that the leased IP addresses for the vms are not returned immediately when the vms are terminated.
I am assuming that this kind of error and stacktrace in the log would be considered a bug even if it does not cause any obvious misbehavior immediately.
I used the current version of devstack but with a modified stack.sh from last week. I modified it to log to files in /var/log/nova instead of stdout so you have to create that directory before running. I used one compute node running api,network,compute and one controller node running everything. After running the modified stack.sh I exported the shell variables in exercise.sh, modified the min count in nova client to create 10 vms, and did
nova boot --flavor 1 --image 3 testvms
In this case the vms seen to have started but there is the AMQP error stacktrace in the log. When this happens, sometimes all the vms come up and sometimes not. I have attached:
1. nova-network log with stacktrace
2. modified stack.sh
3. localrc for both controller and compute
4. nova.conf (even though it is generated)
(nova.rpc): TRACE: Traceback (most recent call last): nova/nova/ rpc/impl_ kombu.py" , line 628, in _process_data nova/nova/ rpc/impl_ kombu.py" , line 673, in reply self.msg_ id, *args, **kwargs) nova/nova/ rpc/impl_ kombu.py" , line 781, in msg_reply send(msg_ id, msg) nova/nova/ rpc/impl_ kombu.py" , line 562, in __exit__ nova/nova/ rpc/impl_ kombu.py" , line 547, in _done .reset( ) nova/nova/ rpc/impl_ kombu.py" , line 382, in reset close() lib/python2. 7/dist- packages/ kombu/transport /pyamqplib. py", line 201, in close python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 194, in close python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 97, in wait method( method_ sig, args, content) python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 115, in dispatch_method python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 273, in _close ption: (404, u"NOT_FOUND - no exchange '5b9a19d1b5e849 d7ba749fada648a 375' in vhost '/'", (60, 40), 'Channel. basic_publish' ) d7ba749fada648a 375' in vhost '/'", (60, 40), 'Channel. basic_publish' ) to caller nova/nova/ rpc/impl_ kombu.py" , line 628, in _process_data\n ctxt.reply(None, None)\n', ' File "/opt/stack/ nova/nova/ rpc/impl_ kombu.py" , line 673, in reply\n msg_reply( self.msg_ id, *args, **kwargs)\n', ' File "/opt/stack/ nova/nova/ rpc/impl_ kombu.py" , line 781, in msg_reply\n conn.direct_ send(msg_ id, msg)\n', ' File "/opt/stack/ nova/nova/ rpc/impl_ kombu.py" , line 562, in __exit__\n self._done()\n', ' File "/opt/stack/ nova/nova/ rpc/impl_ kombu.py" , line 547, in _done\n self.connection .reset( )\n', ' File "/opt/stack/ nova/nova/ rpc/impl_ kombu.py" , line 382, in reset\n self.channel. close() \n', ' File "/usr/local/ lib/python2. 7/dist- packages/ kombu/transport /pyamqplib. py", line 201, in close\n super(Channel, self).close()\n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 194, in close\n (20, 41), # Channel. close_ok\ n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 97, in wait\n return self.dispatch_ method( method_ sig, args, content)\n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 115, in dispatch_method\n return amqp_method(self, args)\n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 273, in _close\n (class_id, method_id))\n', 'AMQPChannelExc eption: (404, u"NOT_FOUND - no exchange \'5b9a19d1b5e84 9d7ba749fada648 a375\' in vhost \'/\'", (60, 40), \'Channel. basic_publish\ ')\n']
(nova.rpc): TRACE: File "/opt/stack/
(nova.rpc): TRACE: ctxt.reply(None, None)
(nova.rpc): TRACE: File "/opt/stack/
(nova.rpc): TRACE: msg_reply(
(nova.rpc): TRACE: File "/opt/stack/
(nova.rpc): TRACE: conn.direct_
(nova.rpc): TRACE: File "/opt/stack/
(nova.rpc): TRACE: self._done()
(nova.rpc): TRACE: File "/opt/stack/
(nova.rpc): TRACE: self.connection
(nova.rpc): TRACE: File "/opt/stack/
(nova.rpc): TRACE: self.channel.
(nova.rpc): TRACE: File "/usr/local/
(nova.rpc): TRACE: super(Channel, self).close()
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: (20, 41), # Channel.close_ok
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: return self.dispatch_
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: return amqp_method(self, args)
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: (class_id, method_id))
(nova.rpc): TRACE: AMQPChannelExce
(nova.rpc): TRACE:
2011-10-20 10:45:24,724 ERROR nova.rpc [-] Returning exception (404, u"NOT_FOUND - no exchange '5b9a19d1b5e849
2011-10-20 10:45:24,724 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/opt/stack/