Comment 17 for bug 1888395

Revision history for this message
Vern Hart (vern) wrote : Re: shared live migration of a vm with a vif is broken in train

I upgraded nova-compute nodes to test live migration patch
 - HTTPS_PROXY=http://172.31.254.9:8080/ sudo -E apt-add-repository ppa:billy-olsen/lp1888395-train
 - sudo apt install nova-api-metadata nova-common nova-compute nova-compute-kvm nova-compute-libvirt python3-nova
 - sudo systemctl restart nova-*

Created VM and tried to live-migrate it
 - openstack server create --image auto-sync/ubuntu-xenial-16.04-amd64-server-20200922-disk1.img --network VN_Red --flavor g1t1.small --boot-from-volume 3 --availability-zone zone1 vern1
 - openstack server migrate vern1 --live-migration
 - nova.exception.InternalError: Failure running os_vif plugin unplug method: Failed to unplug VIF VIFGeneric(active=True,address=02:5b:d4:23:c6:b1,has_traffic_filtering=True,id=5bd423c6-b16d-4678-a05a-3ab94af82d4a,network=Network(e593a2d7-f9e0-4038-957a-1378977bc314),plugin='vrouter',port_profile=<?>,preserve_on_delete=False,vif_name='tap5bd423c6-b1'). Got error: Error during the call to vrouter-port-control: ('vrouter-port-control', '--oper=delete', '--uuid=5bd423c6-b16d-4678-a05a-3ab94af82d4a')

Upgraded nova-cloud-controller units as well
 - HTTPS_PROXY=http://172.31.254.9:8080/ sudo -E apt-add-repository ppa:billy-olsen/lp1888395-train
 - sudo apt install nova-api-os-compute nova-common nova-conductor nova-scheduler nova-spiceproxy python3-nova
 - sudo systemctl restart nova-* apache2

Retrying the above create and migrate commands
 - Same error.

Full trace from `openstack server show vern1 -f value -c fault | sed 's/\\n/\n/g;s/\\'\''/'\''/g'`:

{'code': 500, 'created': '2020-10-27T06:16:17Z', 'message': "Failure running os_vif plugin unplug method: Failed to unplug VIF VIFGeneric(active=True,address=02:26:27:d1:5a:36,has_traffic_filtering=True,id=2627d15a-3604-4a11-80d7-12b92ed9cea6,network=Network(e593a2d7-f9e0-4038-957a-1378977bc314),plugin='vrouter',po", 'details': 'Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 110, in unplug
    plugin.unplug(vif, instance_info)
  File "/usr/lib/python3/dist-packages/vif_plug_vrouter/vrouter.py", line 301, in unplug
    self._vrouter_port_delete(instance_info, vif)
  File "/usr/lib/python3/dist-packages/vif_plug_vrouter/vrouter.py", line 293, in _vrouter_port_delete
    vhostuser_socket, vhostuser_mode)
  File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 245, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 204, in remote_call
    raise exc_type(*result[2])
vif_plug_vrouter.exception.VrouterPortControlError: Error during the call to vrouter-port-control: ('vrouter-port-control', '--oper=delete', '--uuid=2627d15a-3604-4a11-80d7-12b92ed9cea6')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 820, in _unplug_os_vif
    os_vif.unplug(vif, instance_info)
  File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 115, in unplug
    raise os_vif.exception.UnplugException(vif=vif, err=err)
os_vif.exception.UnplugException: Failed to unplug VIF VIFGeneric(active=True,address=02:26:27:d1:5a:36,has_traffic_filtering=True,id=2627d15a-3604-4a11-80d7-12b92ed9cea6,network=Network(e593a2d7-f9e0-4038-957a-1378977bc314),plugin='vrouter',port_profile=<?>,preserve_on_delete=False,vif_name='tap2627d15a-36'). Got error: Error during the call to vrouter-port-control: ('vrouter-port-control', '--oper=delete', '--uuid=2627d15a-3604-4a11-80d7-12b92ed9cea6')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 207, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7546, in _post_live_migration
    unplug_nw_info)
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9489, in post_live_migration_at_source
    self.unplug_vifs(instance, network_info)
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1152, in unplug_vifs
    self._unplug_vifs(instance, network_info, False)
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1146, in _unplug_vifs
    self.vif_driver.unplug(instance, vif)
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 844, in unplug
    self._unplug_os_vif(instance, vif_obj)
  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 824, in _unplug_os_vif
    raise exception.InternalError(msg)
nova.exception.InternalError: Failure running os_vif plugin unplug method: Failed to unplug VIF VIFGeneric(active=True,address=02:26:27:d1:5a:36,has_traffic_filtering=True,id=2627d15a-3604-4a11-80d7-12b92ed9cea6,network=Network(e593a2d7-f9e0-4038-957a-1378977bc314),plugin='vrouter',port_profile=<?>,preserve_on_delete=False,vif_name='tap2627d15a-36'). Got error: Error during the call to vrouter-port-control: ('vrouter-port-control', '--oper=delete', '--uuid=2627d15a-3604-4a11-80d7-12b92ed9cea6')
'}