Block Migration doesn't work: Nova searches for the Instance on the destination Compute host

Bug #1044237 reported by Rohit Karajgi
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Jian Wen

Bug Description

I am trying to perform block migration using a recent Devstack installation, with Nova folsom-rc1
but it's not working.
Commit ID: commit 4adbb96b5530184e3f42047a7416d6f315a14150

Issue: Nova searches for the instance to migrate, but on the destination host, and fails as it could not be found.

Here's my setup:
--------------------------

Host 1: ubuntu-devstack1 - Destination host
Controller node running Nova, Keystone, Quantum and Glance (n-api, n-cpu, n-sch, q-agt, q-svc, g-api, g-reg, key)

Host 2: ubuntu-devstack2 - Source host
Compute node running Compute and Quantum Agent (n-cpu, q-agt)

Both hosts are Ubuntu Oneric, time synced, and resolvable to each other using hostnames.

Steps:
1. Spawn an instance - Scheduler spawns it successfully on Host 2 (ubuntu-devstack2), and stays ACTIVE.
2. Perform block migration.

rohit@ubuntu-devstack1:~/devstack$ nova live-migration --block_migrate fdd7f23e-427d-478d-af61-c7bd6c9d6b91 ubuntu-devstack1
(Here, my source=ubuntu-devstack2, and destination=ubuntu-devstack1)

ERROR: Live migration of instance fdd7f23e-427d-478d-af61-c7bd6c9d6b91 to host ubuntu-dev-stack-vertex failed (HTTP 400) (Request-ID: req-a13bf2ad-8be7-43b8-9a07-441c18e7009d)

3. Compute log on source - No logs generated after running the command
4. Compute log on destination - http://paste.openstack.org/show/20593/

It seems nova is looking for the instance on the destination host.

A check for 'virsh list' on both source and destination showed the correct output, instance was listed on the source (ubuntu-devstack2) and did not list on the destination (ubuntu-devstack1).

Jian Wen (wenjianhn)
Changed in nova:
assignee: nobody → Jian Wen (wenjianhn)
assignee: Jian Wen (wenjianhn) → Sina Web Service Dev (sws)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/12784

Changed in nova:
assignee: Sina Web Service Dev (sws) → Jian Wen (wenjianhn)
Thierry Carrez (ttx)
Changed in nova:
importance: Undecided → High
Changed in nova:
milestone: none → folsom-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/12784
Committed: http://github.com/openstack/nova/commit/b006e6bcb4b58039663b5ee0d0b007cf42245e49
Submitter: Jenkins
Branch: master

commit b006e6bcb4b58039663b5ee0d0b007cf42245e49
Author: Jian Wen <email address hidden>
Date: Tue Sep 11 20:57:14 2012 +0800

    libvirt: Fix live block migration

    Current check_can_live_migrate_destination trys to get instance info
    by call self._conn.lookupByName(instance_name) on the destination
    host before migrating. Because the instance is not on the destination
    host yet, it failes with InstanceNotFound.

    This fix gets the available disk size of the destination side.
    On the source side, checks whether the above space is enough before
    migrating.

    Fixes bug 1044237

    Change-Id: I20f64e1f85828049b697a4b1173bac8e5779d45a

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-rc1 → 2012.2
Revision history for this message
neeraj (neerajb146b) wrote :

hello

the above still persist.
i am getting the same http 400 error

Revision history for this message
darkyat (darkyat) wrote :

This bug still appears in Ubuntu 12.04 LTS using OpenStack Havana 2013.2 and completely prevents me from using live-migration.

tags: added: live-migration
Revision history for this message
Dan Reif (a-launchpad20130227) wrote :

This was so impossibly annoying that I have to share the solution I found. In my case, it ended up being that nova.conf was not owned by nova. This is absurd, because the mode was 644, so it shouldn't have mattered in the slightest. Nonetheless, after over a week debugging this (going so far as to add custom debug code, strace'ing relevant processes, doing an md5sum comparison of a working host and a non-working one; I pulled out all the stops!), this simple change fixed it for me instantly.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.