Comment 5 for bug 855030

Looks likeyou are running out of ips. Maybe you are running /27 which only has 32 addresses?

On Sep 21, 2011, at 6:22 AM, David Kranz wrote:

> ii python-kombu 1.0.4-2 AMQP Messaging Framework for Python
> ii rabbitmq-server 2.5.0-1ubuntu2 An AMQP server written in Erlang
>
> I added the carrot suggestion to nova.conf and it still fails about the
> same 1 in 20 or so vms not coming up but with a different error. I would
> be happy to help diagnose this if you have any other suggestions.
>
>
>
> 2011-09-21 09:17:24,546 DEBUG nova.network.manager [db549763-93f2-444f-bdd8-d4931b2fa6c7 6deb27e2-4b73-4d73-8196-6ccdbcee6859 kranzproje
> ct] network allocations for instance 108 from (pid=1096) allocate_for_instance /usr/lib/pymodules/python2.7/nova/network/manager.py:431
> 2011-09-21 09:17:24,705 ERROR nova.rpc [-] Exception during message handling
> (nova.rpc): TRACE: Traceback (most recent call last):
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_carrot.py", line 285, in _process_data
> (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/network/manager.py", line 216, in allocate_for_instance
> (nova.rpc): TRACE: ips = super(FloatingIP, self).allocate_for_instance(context, **kwargs)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/network/manager.py", line 438, in allocate_for_instance
> (nova.rpc): TRACE: requested_networks=requested_networks)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/network/manager.py", line 166, in _allocate_fixed_ips
> (nova.rpc): TRACE: vpn=vpn, address=address)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/network/manager.py", line 609, in allocate_fixed_ip
> (nova.rpc): TRACE: instance_id)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/db/api.py", line 343, in fixed_ip_associate_pool
> (nova.rpc): TRACE: instance_id, host)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/db/sqlalchemy/api.py", line 101, in wrapper
> (nova.rpc): TRACE: return f(*args, **kwargs)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/db/sqlalchemy/api.py", line 720, in fixed_ip_associate_pool
> (nova.rpc): TRACE: raise exception.NoMoreFixedIps()
> (nova.rpc): TRACE: NoMoreFixedIps: None
> (nova.rpc): TRACE:
> 2011-09-21 09:17:24,739 ERROR nova.rpc [-] Returning exception None to caller
> 2011-09-21 09:17:24,739 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/855030
>
> Title:
> Encountering sporadic AMQPChannelException
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Running one controller and one compute, using nova-network version
> 2011.3~rc~20110909.r1155-0ub on Oneiric . Repeating the following:
>
> Create four instances, terminate them.
>
> This mostly works but one in every three or four times, one of the vms
> does not come up and I see this error in the nova-network log. This
> can occur both in the controller node (also running compute) or the
> compute node. (contents of nova.conf follows)
>
> 2011-09-20 13:22:59,295 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=1082) inner /usr/lib/pymodules/python2.7/nova/utils.py:672
> 2011-09-20 13:22:59,295 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=1082) inner /usr/lib/pymodules/python2.7/nova/utils.py:677
> 2011-09-20 13:22:59,296 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=1082) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
> 2011-09-20 13:22:59,311 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1082) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
> 2011-09-20 13:22:59,350 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=1082) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
> 2011-09-20 13:22:59,366 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1082) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
> 2011-09-20 13:22:59,424 ERROR nova.rpc [-] Exception during message handling
> (nova.rpc): TRACE: Traceback (most recent call last):
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 628, in _process_data
> (nova.rpc): TRACE: ctxt.reply(None, None)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 673, in reply
> (nova.rpc): TRACE: msg_reply(self.msg_id, *args, **kwargs)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 781, in msg_reply
> (nova.rpc): TRACE: conn.direct_send(msg_id, msg)
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 562, in __exit__
> (nova.rpc): TRACE: self._done()
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 547, in _done
> (nova.rpc): TRACE: self.connection.reset()
> (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 382, in reset
> (nova.rpc): TRACE: self.channel.close()
> (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 194, in close
> (nova.rpc): TRACE: (20, 41), # Channel.close_ok
> (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
> (nova.rpc): TRACE: return self.dispatch_method(method_sig, args, content)
> (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method
> (nova.rpc): TRACE: return amqp_method(self, args)
> (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close
> (nova.rpc): TRACE: (class_id, method_id))
> (nova.rpc): TRACE: AMQPChannelException: (404, u"NOT_FOUND - no exchange '3ff1ba7e274a4ec2a2b0a217d7532c70' in vhost '/'", (60, 40), 'Channel.basic_publish')
> (nova.rpc): TRACE:
> 2011-09-20 13:22:59,451 ERROR nova.rpc [-] Returning exception (404, u"NOT_FOUND - no exchange '3ff1ba7e274a4ec2a2b0a217d7532c70' in vhost '/'", (60, 40), 'Channel.basic_publish') to caller
> 2011-09-20 13:22:59,452 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 628, in _process_data\n ctxt.reply(None, None)\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 673, in reply\n msg_reply(self.msg_id, *args, **kwargs)\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 781, in msg_reply\n conn.direct_send(msg_id, msg)\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 562, in __exit__\n self._done()\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 547, in _done\n self.connection.reset()\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 382, in reset\n self.channel.close()\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 194, in close\n (20, 41), # Channel.close_ok\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait\n return self.dispatch_method(method_sig, args, content)\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method\n return amqp_method(self, args)\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close\n (class_id, method_id))\n', 'AMQPChannelException: (404, u"NOT_FOUND - no exchange \'3ff1ba7e274a4ec2a2b0a217d7532c70\' in vhost \'/\'", (60, 40), \'Channel.basic_publish\')\n']
>
>
> I also see this at the same time in the rabbitmq log:
>
> =ERROR REPORT==== 20-Sep-2011::13:22:59 ===
> connection <0.1061.0>, channel 1 - error:
> {amqp_error,not_found,
> "no exchange '3ff1ba7e274a4ec2a2b0a217d7532c70' in vhost '/'",
> 'basic.publish'}
>
>
>
> --flagfile=/etc/nova/nova-compute.conf
> --use_deprecated_auth
> --dhcpbridge_flagfile=/etc/nova/nova.conf
> --dhcpbridge=/usr/bin/nova-dhcpbridge
> --sql_connection=mysql://nova:notnova@172.18.0.141/nova
> --s3_host=172.18.0.141
> --rabbit_host=172.18.0.141
> --glance_api_servers=172.18.0.141:9292
> --logdir=/var/log/nova
> --state_path=/var/lib/nova
> --lock_path=/var/lock/nova
> --verbose
> --ec2_url=http://172.18.0.141:8773/services/Cloud
> --dmz_cidr=172.18.0.141/32
> --fixed_range=172.18.77.0/27
> --network_size=8
> --flat_network_dhcp_start=172.18.7.6
> --image_service=nova.image.glance.GlanceImageService
> --bridge_interface=eth1
> --flat_network_bridge=br100
> --network_manager=nova.network.manager.FlatDHCPManager
> --public_interface=eth0
> --multi_host
> --osapi_host=172.18.0.141
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/855030/+subscriptions