nova-live-migration job fails during evacuate negative test

Looks like libvirtd restarted allowing the negative evacuation test to result in an ACTIVE instance on the controller. The current task stops the following service and associated sockets:

roles/run-evacuate-hook/tasks/main.yaml

   34 - name: Stop libvirtd on "{{ inventory_hostname }}"
   35 become: true
   36 systemd:
   37 ¦ name: "{{ item }}"
   38 ¦ state: stopped
   39 with_items:
   40 ¦ - libvirtd.service
   41 ¦ - libvirtd.socket
   42 ¦ - libvirtd-admin.socket
   43 ¦ - libvirtd-ro.socket

I guess I'm missing something here?

Fix proposed to branch: master
Review: https://review.opendev.org/762623

Can it be related to https://review.opendev.org/c/openstack/nova/+/754815?

What we have here:

1. booting a VM on SUBNODE_HOSTNAME and disabling compute service on SUBNODE_HOSTNAME
openstack --os-compute-api-version 2.74 server create --image ec033028-30f6-497d-936a-4bea98cdafea --flavor 1 --nic net-id=209ff518-ff9f-46bb-ac0a-a20e76b35daa --host ubuntu-focal-rax-iad-0021734807 --wait evacuate-test

...

2. Evacuating that VM to CONTROLLER_HOSTNAME
nova --os-compute-api-version 2.67 evacuate --force evacuate-test ubuntu-focal-rax-iad-0021734805
where it actually becomes "active" instead of going to "error".
In case of success, it should fail on controlller with error "nova.exception.HypervisorUnavailable: Connection to the hypervisor is broken on host".

But indeed libvirtd was started again:

2020-11-12 08:44:00.128+0000: 102594: info : libvirt version: 6.0.0, package: 0ubuntu8.5 (Christian Ehrhardt <email address hidden> Thu, 08 Oct 2020 07:36:06 +0200)
2020-11-12 08:44:00.128+0000: 102594: info : hostname: ubuntu-focal-rax-iad-0021734805
2020-11-12 08:44:00.128+0000: 102594: debug : virNetlinkStartup:124 : Running global netlink initialization

Just before 'Run negative evacuate tests':

2020-11-12 08:44:02.021505 | TASK [run-evacuate-hook : Run negative evacuate tests]

Ansible report:

2020-11-12 08:43:59.147000 | LOOP [run-evacuate-hook : Stop libvirtd on "controller"]
2020-11-12 08:44:00.097319 | controller | changed: "libvirtd.service"
2020-11-12 08:44:00.099003 | controller | changed: All items complete
2020-11-12 08:44:00.099120 |
2020-11-12 08:44:00.714245 | controller | changed: "libvirtd.socket"
2020-11-12 08:44:01.298388 | controller | ok: "libvirtd-admin.socket"
2020-11-12 08:44:01.872071 | controller | ok: "libvirtd-ro.socket"

Syslog report:

Nov 12 08:43:59 ubuntu-focal-rax-iad-0021734805 sudo[102576]: zuul : TTY=unknown ; PWD=/home/zuul ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-ikljghyeytpdnrvynsdgogdmzisxdquq ; /usr/bin/python3
Nov 12 08:43:59 ubuntu-focal-rax-iad-0021734805 sudo[102576]: pam_unix(sudo:session): session opened for user root by (uid=0)
Nov 12 08:43:59 ubuntu-focal-rax-iad-0021734805 sudo[102576]: pam_unix(sudo:session): session closed for user root
Nov 12 08:44:00 ubuntu-focal-rax-iad-0021734805 sudo[102596]: zuul : TTY=unknown ; PWD=/home/zuul ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-qdjdltsjeqgfxczzgvknhrqebzarfpca ; /usr/bin/python3
Nov 12 08:44:00 ubuntu-focal-rax-iad-0021734805 sudo[102596]: pam_unix(sudo:session): session opened for user root by (uid=0)
Nov 12 08:44:00 ubuntu-focal-rax-iad-0021734805 sudo[102596]: pam_unix(sudo:session): session closed for user root
Nov 12 08:44:00 ubuntu-focal-rax-iad-0021734805 sudo[102682]: zuul : TTY=unknown ; PWD=/home/zuul ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-bxmxlpjmzyjadkbullovdfrbmiqvbibg ; /usr/bin/python3
Nov 12 08:44:00 ubuntu-focal-rax-iad-0021734805 sudo[102682]: pam_unix(sudo:session): session opened for user root by (uid=0)
Nov 12 08:44:01 ubuntu-focal-rax-iad-0021734805 kernel: iptables dropped: IN=eth0 OUT= MAC=bc:76:4e:20:17:0d:e4:c7:22:60:5a:c1:08:00 SRC=45.146.165.152 DST=104.130.239.160 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=32541 DF PROTO=TCP SPT=38250 DPT=3389...

libvirtd.service has Restart=on-failure. This can be a reason libvirtd started immediately after stop.

This issue was fixed in the openstack/nova 23.0.0.0rc1 release candidate.