live-migration tempest test failed because the instance didn't seem to migrate to its target host

Bug #1779270 reported by Masayuki Igawa
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
New
Undecided
Unassigned

Bug Description

Description
===========
test_live_block_migration[0][1][2] tempest tests failed frequently[3].

[0] http://status.openstack.org/openstack-health/#/test/tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration?duration=P1M
[1] http://status.openstack.org/openstack-health/#/test/tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration?duration=P1M
[2] http://status.openstack.org/openstack-health/#/test/tempest.api.compute.admin.test_live_migration.LiveMigrationRemoteConsolesV26Test.test_live_block_migration?duration=P1M
[3] http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Live%20Migration%20failed.%20Migrations%20list%20for%20Instance%20%5C%22

Steps to reproduce
==================
* run nova-live-migration, neutron-tempest-multinode-full or some other job which contains the test_live_block_migration test.

Expected result
===============
* The test doesn't fail. The instance is migrated.

Actual result
=============
* The instance doesn't seem to be migrated, and then, the test fails.

Environment
===========
* In the gate, master branch. Not sure about the stable branches.

Logs & Configs
==============
Quote from http://logs.openstack.org/34/575734/8/gate/nova-live-migration/cb90e1d/job-output.txt.gz

------------
2018-06-25 19:47:23.645249 | primary | 2018-06-25 19:47:23.643 | {1} tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration [25.911015s] ... FAILED
2018-06-25 19:47:23.647327 | primary | 2018-06-25 19:47:23.646 |
2018-06-25 19:47:23.653059 | primary | 2018-06-25 19:47:23.648 | Captured traceback:
2018-06-25 19:47:23.655235 | primary | 2018-06-25 19:47:23.654 | ~~~~~~~~~~~~~~~~~~~
2018-06-25 19:47:23.657181 | primary | 2018-06-25 19:47:23.656 | Traceback (most recent call last):
2018-06-25 19:47:23.658951 | primary | 2018-06-25 19:47:23.658 | File "tempest/api/compute/admin/test_live_migration.py", line 127, in test_live_block_migration
2018-06-25 19:47:23.664233 | primary | 2018-06-25 19:47:23.663 | self._test_live_migration()
2018-06-25 19:47:23.669330 | primary | 2018-06-25 19:47:23.668 | File "tempest/api/compute/admin/test_live_migration.py", line 117, in _test_live_migration
2018-06-25 19:47:23.671351 | primary | 2018-06-25 19:47:23.670 | self._live_migrate(server_id, destination_host, state, volume_backed)
2018-06-25 19:47:23.673324 | primary | 2018-06-25 19:47:23.672 | File "tempest/api/compute/admin/test_live_migration.py", line 91, in _live_migrate
2018-06-25 19:47:23.676421 | primary | 2018-06-25 19:47:23.675 | msg)
2018-06-25 19:47:23.678566 | primary | 2018-06-25 19:47:23.678 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
2018-06-25 19:47:23.680280 | primary | 2018-06-25 19:47:23.679 | self.assertThat(observed, matcher, message)
2018-06-25 19:47:23.682290 | primary | 2018-06-25 19:47:23.681 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
2018-06-25 19:47:23.685235 | primary | 2018-06-25 19:47:23.683 | raise mismatch_error
2018-06-25 19:47:23.688145 | primary | 2018-06-25 19:47:23.687 | testtools.matchers._impl.MismatchError: !=:
2018-06-25 19:47:23.690041 | primary | 2018-06-25 19:47:23.689 | reference = u'ubuntu-xenial-ovh-gra1-0000332527'
2018-06-25 19:47:23.691611 | primary | 2018-06-25 19:47:23.691 | actual = u'ubuntu-xenial-ovh-gra1-0000332530'
2018-06-25 19:47:23.693251 | primary | 2018-06-25 19:47:23.692 | : Live Migration failed. Migrations list for Instance 607cb757-f381-44b4-aff9-5cafe6d2cdfe: [
2018-06-25 19:47:23.695192 | primary | 2018-06-25 19:47:23.694 | {u'id': 9, u'instance_uuid': u'607cb757-f381-44b4-aff9-5cafe6d2cdfe', u'status': u'error', u'updated_at': u'2018-06-25T19:47:23.000000', u'new_instance_type_id': 11, u'old_instance_type_id': 11, u'source_node': u'ubuntu-xenial-ovh-gra1-0000332530', u'dest_host': None, u'source_compute': u'ubuntu-xenial-ovh-gra1-0000332530', u'dest_node': u'ubuntu-xenial-ovh-gra1-0000332527', u'created_at': u'2018-06-25T19:47:11.000000', u'dest_compute': u'ubuntu-xenial-ovh-gra1-0000332527'}]
------------

tags: added: live-migration
Revision history for this message
Matt Riedemann (mriedem) wrote :

The real failure is here:

http://logs.openstack.org/34/575734/8/gate/nova-live-migration/cb90e1d/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_Jun_25_19_47_21_967953

Jun 25 19:47:21.967953 ubuntu-xenial-ovh-gra1-0000332530 nova-compute[4672]: ERROR nova.virt.libvirt.driver [None req-2cf5b040-9ab2-4d6d-b4a6-7d929372305e service nova] [instance: 607cb757-f381-44b4-aff9-5cafe6d2cdfe] Live Migration failure: internal error: qemu unexpectedly closed the monitor: 2018-06-25T19:47:17.020552Z qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
Jun 25 19:47:21.968244 ubuntu-xenial-ovh-gra1-0000332530 nova-compute[4672]: 2018-06-25T19:47:21.867441Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x11e inconsistent with Host index 0x11f: delta 0xffff
Jun 25 19:47:21.968547 ubuntu-xenial-ovh-gra1-0000332530 nova-compute[4672]: 2018-06-25T19:47:21.867473Z qemu-system-x86_64: Failed to load virtio-blk:virtio
Jun 25 19:47:21.968763 ubuntu-xenial-ovh-gra1-0000332530 nova-compute[4672]: 2018-06-25T19:47:21.867482Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-blk'
Jun 25 19:47:21.968969 ubuntu-xenial-ovh-gra1-0000332530 nova-compute[4672]: 2018-06-25T19:47:21.867569Z qemu-system-x86_64: load of migration failed: Operation not permitted: libvirtError: internal error: qemu unexpectedly closed the monitor: 2018-06-25T19:47:17.020552Z qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
Jun 25 19:47:22.491645 ubuntu-xenial-ovh-gra1-0000332530 nova-compute[4672]: ERROR nova.virt.libvirt.driver [None req-2cf5b040-9ab2-4d6d-b4a6-7d929372305e service nova] [instance: 607cb757-f381-44b4-aff9-5cafe6d2cdfe] Migration operation has aborted

Which is a known issue so this is a duplicate bug.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.