Description
===========
The two voting LVM jobs: manila-tempest-plugin-lvm and manila-tempest-minimal-lvm-ipv6-only started failing at the beginning of July 2020. The failures were observed consistently on rax nodes from the following Rackspace regions:
rax-dfw
rax-iad
rax-ord
The cause of failure seems to be because the devstack node reboots in the middle of the job
The observed sequence of operations is:
- A bunch of tests are run, successfully
- A reboot occurs
- manila-share service now cannot connect to the ephemeral vgs that were set up
- further share creation attempts to fail in the scheduler since the share service is down
- testr results are compiled, and all tests are marked as failed
This issue seems to only occur on rax nodes so far, and hasn't occurred on any of the other cloud providers that Zuul could schedule the job on, which include Vexxhost, OVH, INAP, etc.
Logs and configuration from one of the failures has been attached to this bug report.
Reducing the test job concurrency did not resolve the issue, nor did reducing the LVM driver's backing file size. These attempts have been made here:
https://review.opendev.org/#/c/739775/5 (concurrency was set to 1, reboot still occurred)
https://review.opendev.org/#/c/740109/ (backing file size was halved, reboot still occurred)
At present, with scenario tests disabled, we're not seeing reboots - it's likely something in the spinning up of virtual machines, and providing LVM shares that's affecting the test node.
One difference between RAX and other providers is that RAX uses Xen hypervisors while the others use KVM.
Might temporarily disable scenario tests to get the gate working again, and bump this issue down to "High":
https:/
https:/