kvm: ssh commands during resize instance gets stuck up

Bug #975035 reported by Unmesh Gurjar
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

Scenario:
I have an OpenStack set up (using master branch) with 2 Compute nodes, following are the Nova configurations details:
    allow_resize_to_same_host=True
    compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
    connection_type=libvirt
    libvirt_type=kvm

Note: There is no password less ssh connection setup between the two Compute nodes.

To reproduce this issue, spawn an instance, once it comes up, resize it. I used the nova CLI command to resize (nova resize d24f03d8-5757-453d-a025-1e7b36c6ef8f 2).

Branch: master

Expected Behavior: The instance should get resized.

Actual Behavior: The instance gets stuck in the following state: vm_state=resizing, task_state=resize_prep, power_state=0

On further analysis, it seems that the source Compute host has destroyed the instance and has got stuck in an 'ssh' command (in migrate_disk_and_power_off( ) method in libvirt/connection.py). Following is the code snippet:

            if same_host:
                utils.execute('mkdir', '-p', inst_base)
            else:
                utils.execute('ssh', dest, 'mkdir', '-p', inst_base)

Effectively, the instance directory on the source Compute host has the 'instance-00000003_resize' directory.
To proceed further, I killed the blocking 'ssh' command, however, it further got stuck in removing the instance directory on the remote Compute host. Here is the related code snippet:

            try:
                if os.path.exists(inst_base_resize):
                    utils.execute('rm', '-rf', inst_base)
                    utils.execute('mv', inst_base_resize, inst_base)
                    utils.execute('ssh', dest, 'rm', '-rf', inst_base)
            except Exception:
                pass

On killing this ssh command, the instance went into following state:
    vm_state=error, task_state=resize_prep, power_state=0.

Revision history for this message
Russell Bryant (russellb) wrote :

Allowing resize on the same host doesn't it force it to be that way. You would have to customize the scheduler to do that. You just need to set up ssh for this work.

Changed in nova:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.