OpenStack Compute (nova)

RPC timeout with multi_host networking

Bug #1043905 reported by Mate Lakat
22
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Mate Lakat
OpenStack Compute (nova) 2012.2 "folsom"

Bug Description

I am running on devstack on XenServer, and the terminate-instance fails from euca exercise. I also cannot terminate the instance through Horizon.

exercise.sh log:
+ timeout 30 sh -c 'while euca-describe-instances i-00000001 | grep -q running; do sleep 1; done'
+ echo 'server didn'\''t terminate within 30 seconds'
server didn't terminate within 30 seconds
+ exit 1

 Stacktraces:

nova.openstack.common.rpc.common Traceback (most recent call last):
nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 513, in ensure
nova.openstack.common.rpc.common return method(*args, **kwargs)
nova.openstack.common.rpc.common File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 590, in _consume
nova.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)
nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
nova.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)
nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
nova.openstack.common.rpc.common return connection.drain_events(**kwargs)
nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
nova.openstack.common.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
nova.openstack.common.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
nova.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)
nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout
nova.openstack.common.rpc.common return self.method_reader.read_method()
nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
nova.openstack.common.rpc.common raise m
nova.openstack.common.rpc.common timeout: timed out

nova.openstack.common.rpc.amqp Traceback (most recent call last):
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 275, in _process_data
nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 117, in wrapped
nova.openstack.common.rpc.amqp temp_level, payload)
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
nova.openstack.common.rpc.amqp self.gen.next()
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 92, in wrapped
nova.openstack.common.rpc.amqp return f(*args, **kw)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 211, in decorated_function
nova.openstack.common.rpc.amqp pass
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
nova.openstack.common.rpc.amqp self.gen.next()
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function
nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 184, in decorated_function
nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 237, in decorated_function
nova.openstack.common.rpc.amqp instance_uuid, e, sys.exc_info())
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
nova.openstack.common.rpc.amqp self.gen.next()
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 226, in decorated_function
nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 955, in terminate_instance
nova.openstack.common.rpc.amqp do_terminate_instance(instance)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/utils.py", line 733, in inner
nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 947, in do_terminate_instance
nova.openstack.common.rpc.amqp self._delete_instance(context, instance)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 917, in _delete_instance
nova.openstack.common.rpc.amqp self._shutdown_instance(context, instance)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 874, in _shutdown_instance
nova.openstack.common.rpc.amqp self._deallocate_network(context, instance)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 800, in _deallocate_network
nova.openstack.common.rpc.amqp self.network_api.deallocate_for_instance(context, instance)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/network/api.py", line 260, in deallocate_for_instance
nova.openstack.common.rpc.amqp 'args': args})
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 102, in call
nova.openstack.common.rpc.amqp return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 712, in call
nova.openstack.common.rpc.amqp rpc_amqp.get_connection_pool(conf, Connection))
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 368, in call
nova.openstack.common.rpc.amqp rv = list(rv)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 329, in __iter__
nova.openstack.common.rpc.amqp self.done()
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
nova.openstack.common.rpc.amqp self.gen.next()
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 326, in __iter__
nova.openstack.common.rpc.amqp self._iterator.next()
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 595, in iterconsume
nova.openstack.common.rpc.amqp yield self.ensure(_error_callback, _consume)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 526, in ensure
nova.openstack.common.rpc.amqp error_callback(e)
nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 576, in _error_callback
nova.openstack.common.rpc.amqp raise rpc_common.Timeout()
nova.openstack.common.rpc.amqp Timeout: Timeout while waiting on RPC response.

Tags: xenserver xapi
Revision history for this message
Mate Lakat (mate-lakat) wrote :

See my comment at:
https://github.com/openstack/nova/commit/32b0346eb40a94ec6def3eed01e9424b0c660c53#nova/network/manager.py

Changed in nova:
assignee: nobody → Mate Lakat (mate-lakat)
Yun Mao (yunmao)
Changed in nova:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/12249

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/12249
Committed: http://github.com/openstack/nova/commit/cda9b490eb601fe846061720c5c66a46dc74048b
Submitter: Jenkins
Branch: master

commit cda9b490eb601fe846061720c5c66a46dc74048b
Author: Mate Lakat <email address hidden>
Date: Fri Aug 31 10:12:26 2012 +0100

    Fix deallocate_fixed_ip invocation

    Fixes bug 1043905.

    The host parameter was not passed to deallocate_fixed_ip method, so that
    the deallocation tried to do a remote call to another host, which timed
    out, thus instance termination failed.

    Change-Id: I18d6de31d15767a45f389ae14769985c3951d560

Changed in nova:
status: In Progress → Fix Committed
Mark McLoughlin (markmc)
summary: - instance fails to terminate (XAPI)
+ RPC timeout with multi_host networking
Revision history for this message
Mate Lakat (mate-lakat) wrote :

Guys, while we are here. I fixed only the deallocate_for_instance call in NetworkManager, but I think something still smells:

See my comment in the original commit, at line 1738:

https://github.com/openstack/nova/commit/32b0346eb40a94ec6def3eed01e9424b0c660c53#L0R1738

So if you are using FlatManager, and multi_host, you are likely to run into some similar problem, as the host is not passed through.

Thierry Carrez (ttx)
Changed in nova:
milestone: none → folsom-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-rc1 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.