Comment 11 for bug 1649586

There is one thing I don't understand. There are a lot of errors that AMQP server is unreachable:

http://logs.openstack.org/04/409904/2/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/8e63e9c/logs/subnode-2/old/screen-n-cpu.txt.gz#_2016-12-13_14_09_18_649

2016-12-13 14:09:14.551 12173 DEBUG oslo_messaging._drivers.amqpdriver [req-b71314d0-c5c2-44b3-927f-072a97636c47 - -] CAST unique_id: ede0ffc923d64e9a8502b1fe772c2dcd FANOUT topic 'scheduler' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:437
2016-12-13 14:09:18.649 12173 ERROR oslo.messaging._drivers.impl_rabbit [req-070105f7-302c-4ea2-ba6c-6ed1f21cea97 tempest-LiveBlockMigrationTestJSON-1846422298 tempest-LiveBlockMigrationTestJSON-1846422298] [f253dd7c-f063-4a70-8267-9551112a3939] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None
2016-12-13 14:09:19.291 12173 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: f9b691fb9de14ce29b59b339e0c2f355 exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448

However, nova-compute accepts requests through RPC, e.g., nova spawned new instance at 14:10:46.907

http://logs.openstack.org/04/409904/2/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/8e63e9c/logs/subnode-2/old/screen-n-cpu.txt.gz#_2016-12-13_14_10_46_907

2016-12-13 14:10:46.907 12173 INFO nova.virt.libvirt.driver [-] [instance: 9c4a72aa-0540-470d-9446-969a7db638e7] Instance spawned successfully.
2016-12-13 14:10:46.907 12173 INFO nova.compute.manager [req-5f7a2bde-892a-46a3-910c-134443029483 tempest-LiveAutoBlockMigrationV225TestJSON-970444200 tempest-LiveAutoBlockMigrationV225TestJSON-970444200] [instance: 9c4a72aa-0540-470d-9446-969a7db638e7] Took 1.48 seconds to spawn the instance on the hypervisor.

But it never reconnected to RPC after error at 14:10:30.682

http://logs.openstack.org/04/409904/2/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/8e63e9c/logs/subnode-2/old/screen-n-cpu.txt.gz#_2016-12-13_14_10_30_682

2016-12-13 14:10:30.682 12173 ERROR oslo.messaging._drivers.impl_rabbit [req-070105f7-302c-4ea2-ba6c-6ed1f21cea97 tempest-LiveBlockMigrationTestJSON-1846422298 tempest-LiveBlockMigrationTestJSON-1846422298] [f253dd7c-f063-4a70-8267-9551112a3939] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None