Instance becomes stuck in 'Networking' state due to nova-compute error

Bug #772681 reported by Janet Post
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Bharath Kumar Kobagana

Bug Description

HostedOSCC1# euca-run-instances ami-335e2dd1 -k ami-335e2dd1

Instance became stuck in a networking state on the remote host HostedOS1.

Nova Version: 2011.3~bzr1033-0ubuntu0ppa1~lucid1

Nova config:
root@HostedOS1:/var/log/nova# cat /etc/nova/nova.conf
--daemonize=1
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose=1
--sql_connection=mysql://root:nova@192.168.12.59/nova
--s3_host=192.168.12.59
--rabbit_host=192.168.12.59
--cc_host=192.168.12.59
--ec2_url=http://192.168.12.59:8773/services/Cloud
--network_manager=nova.network.manager.FlatDHCPManager
--fixed_range=192.168.2.48/28
--network_size=13
--flat_network_dhcp_start=192.168.2.50

Log:

nova-compute.log-2011-04-28 10:10:53,936 DEBUG nova.rpc [-] received {u'_context_request_id': u'CY6GJ4RK7C597B72E04-', u'_context_read_deleted': False, u'args': {u'instance_id': 25, u'injected_files': None, u'availability_zone': None}, u'_context_is_admin': True, u'_context_timestamp': u'2011-04-28T14:10:53Z', u'_context_user': u'root', u'method': u'run_instance', u'_context_project': u'testproject', u'_context_remote_address': u'192.168.12.59'} from (pid=27312) _receive /usr/lib/pymodules/python2.6/nova/rpc.py:167
nova-compute.log-2011-04-28 10:10:53,937 DEBUG nova.rpc [-] unpacked context: {'timestamp': u'2011-04-28T14:10:53Z', 'remote_address': u'192.168.12.59', 'project': u'testproject', 'is_admin': True, 'user': u'root', 'request_id': u'CY6GJ4RK7C597B72E04-', 'read_deleted': False} from (pid=27312) _unpack_context /usr/lib/pymodules/python2.6/nova/rpc.py:331
nova-compute.log-2011-04-28 10:10:53,994 AUDIT nova.compute.manager [CY6GJ4RK7C597B72E04- root testproject] instance 25: starting...
nova-compute.log-2011-04-28 10:10:54,148 DEBUG nova.rpc [-] Making asynchronous call on network.HostedOSCC1 ... from (pid=27312) call /usr/lib/pymodules/python2.6/nova/rpc.py:350
nova-compute.log-2011-04-28 10:10:54,148 DEBUG nova.rpc [-] MSG_ID is f35e1919189e4938a99f28f0ea77636c from (pid=27312) call /usr/lib/pymodules/python2.6/nova/rpc.py:353
nova-compute.log-2011-04-28 10:10:54,557 ERROR nova [-] Exception during message handling
nova-compute.log-(nova): TRACE: Traceback (most recent call last):
nova-compute.log-(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 188, in _receive
nova-compute.log-(nova): TRACE: rval = node_func(context=ctxt, **node_args)
nova-compute.log-(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
nova-compute.log-(nova): TRACE: return f(*args, **kw)
nova-compute.log-(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 222, in run_instance
nova-compute.log-(nova): TRACE: "vpn": is_vpn}})
nova-compute.log-(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 385, in call
nova-compute.log-(nova): TRACE: raise wait_msg.result
nova-compute.log:(nova): TRACE: RemoteError: TypeError 'NoneType' object is unsubscriptable
nova-compute.log:(nova): TRACE: [u'Traceback (most recent call last):\n', u' File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 188, in _receive\n rval = node_func(context=ctxt, **node_args)\n', u' File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 459, in allocate_fixed_ip\n self.driver.update_dhcp(context, network_ref[\'id\'])\n', u' File "/usr/lib/pymodules/python2.6/nova/utils.py", line 607, in inner\n retval = f(*args, **kwargs)\n', u' File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 564, in update_dhcp\n f.write(get_dhcp_hosts(context, network_id))\n', u' File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 546, in get_dhcp_hosts\n hosts.append(_host_dhcp(fixed_ip_ref))\n', u' File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 654, in _host_dhcp\n return "%s,%s.%s,%s" % (instance_ref[\'mac_address\'],\n', u"TypeError: 'NoneType' object is unsubscriptable\n"]
nova-compute.log-(nova): TRACE:

Revision history for this message
Brian Waldon (bcwaldon) wrote :

Looks like network manager failed to retrieve the instance from the database. Possibly a race condition? Are you still able to reproduce this, Janet?

Changed in nova:
importance: Undecided → Low
status: New → Confirmed
Revision history for this message
Bharath Kumar Kobagana (kobagana-kumar) wrote :

This bug is fixed in Essex version. I tried with:
"euca-run-instances ami-335e2dd1 -k ami-335e2dd1"

Then it creates instance and the instance comes to active state and not raising any error.

Changed in nova:
assignee: nobody → Bharath Kumar Kobagana (kobagana-kumar)
Tom Fifield (fifieldt)
Changed in nova:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.