HostedOSCC1# euca-run-instances ami-335e2dd1 -k ami-335e2dd1
Instance became stuck in a networking state on the remote host HostedOS1.
Nova Version: 2011.3~bzr1033-0ubuntu0ppa1~lucid1
Nova config:
root@HostedOS1:/var/log/nova# cat /etc/nova/nova.conf
--daemonize=1
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose=1
--sql_connection=mysql://root:nova@192.168.12.59/nova
--s3_host=192.168.12.59
--rabbit_host=192.168.12.59
--cc_host=192.168.12.59
--ec2_url=http://192.168.12.59:8773/services/Cloud
--network_manager=nova.network.manager.FlatDHCPManager
--fixed_range=192.168.2.48/28
--network_size=13
--flat_network_dhcp_start=192.168.2.50
Log:
nova-compute.log-2011-04-28 10:10:53,936 DEBUG nova.rpc [-] received {u'_context_request_id': u'CY6GJ4RK7C597B72E04-', u'_context_read_deleted': False, u'args': {u'instance_id': 25, u'injected_files': None, u'availability_zone': None}, u'_context_is_admin': True, u'_context_timestamp': u'2011-04-28T14:10:53Z', u'_context_user': u'root', u'method': u'run_instance', u'_context_project': u'testproject', u'_context_remote_address': u'192.168.12.59'} from (pid=27312) _receive /usr/lib/pymodules/python2.6/nova/rpc.py:167
nova-compute.log-2011-04-28 10:10:53,937 DEBUG nova.rpc [-] unpacked context: {'timestamp': u'2011-04-28T14:10:53Z', 'remote_address': u'192.168.12.59', 'project': u'testproject', 'is_admin': True, 'user': u'root', 'request_id': u'CY6GJ4RK7C597B72E04-', 'read_deleted': False} from (pid=27312) _unpack_context /usr/lib/pymodules/python2.6/nova/rpc.py:331
nova-compute.log-2011-04-28 10:10:53,994 AUDIT nova.compute.manager [CY6GJ4RK7C597B72E04- root testproject] instance 25: starting...
nova-compute.log-2011-04-28 10:10:54,148 DEBUG nova.rpc [-] Making asynchronous call on network.HostedOSCC1 ... from (pid=27312) call /usr/lib/pymodules/python2.6/nova/rpc.py:350
nova-compute.log-2011-04-28 10:10:54,148 DEBUG nova.rpc [-] MSG_ID is f35e1919189e4938a99f28f0ea77636c from (pid=27312) call /usr/lib/pymodules/python2.6/nova/rpc.py:353
nova-compute.log-2011-04-28 10:10:54,557 ERROR nova [-] Exception during message handling
nova-compute.log-(nova): TRACE: Traceback (most recent call last):
nova-compute.log-(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 188, in _receive
nova-compute.log-(nova): TRACE: rval = node_func(context=ctxt, **node_args)
nova-compute.log-(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
nova-compute.log-(nova): TRACE: return f(*args, **kw)
nova-compute.log-(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 222, in run_instance
nova-compute.log-(nova): TRACE: "vpn": is_vpn}})
nova-compute.log-(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 385, in call
nova-compute.log-(nova): TRACE: raise wait_msg.result
nova-compute.log:(nova): TRACE: RemoteError: TypeError 'NoneType' object is unsubscriptable
nova-compute.log:(nova): TRACE: [u'Traceback (most recent call last):\n', u' File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 188, in _receive\n rval = node_func(context=ctxt, **node_args)\n', u' File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 459, in allocate_fixed_ip\n self.driver.update_dhcp(context, network_ref[\'id\'])\n', u' File "/usr/lib/pymodules/python2.6/nova/utils.py", line 607, in inner\n retval = f(*args, **kwargs)\n', u' File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 564, in update_dhcp\n f.write(get_dhcp_hosts(context, network_id))\n', u' File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 546, in get_dhcp_hosts\n hosts.append(_host_dhcp(fixed_ip_ref))\n', u' File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 654, in _host_dhcp\n return "%s,%s.%s,%s" % (instance_ref[\'mac_address\'],\n', u"TypeError: 'NoneType' object is unsubscriptable\n"]
nova-compute.log-(nova): TRACE:
Looks like network manager failed to retrieve the instance from the database. Possibly a race condition? Are you still able to reproduce this, Janet?