swap volume intermittently fails with "libvirtError: block copy still active: domain has active block job"

Bug #1763410 reported by Matt Riedemann
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Expired
Low
Unassigned

Bug Description

Seeing this in some CI runs:

http://logs.openstack.org/67/560467/1/check/nova-next/9ceb996/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_11_16_19_23_843152

Apr 11 16:19:23.843152 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver [None req-4ca5f968-bbce-4826-b16c-cc02d63c07c2 service nova] Failure rebasing volume /dev/sda on vdb.: libvirtError: internal error: unable to execute QEMU command 'block-job-complete': The active block job 'drive-virtio-disk1' cannot be completed
Apr 11 16:19:23.843399 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
Apr 11 16:19:23.843646 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1529, in _swap_volume
Apr 11 16:19:23.843882 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver dev.abort_job(pivot=True)
Apr 11 16:19:23.844165 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 757, in abort_job
Apr 11 16:19:23.844416 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver self._guest._domain.blockJobAbort(self._disk, flags=flags)
Apr 11 16:19:23.844666 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
Apr 11 16:19:23.844905 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver result = proxy_call(self._autowrap, f, *args, **kwargs)
Apr 11 16:19:23.845168 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
Apr 11 16:19:23.845412 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver rv = execute(f, *args, **kwargs)
Apr 11 16:19:23.845659 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
Apr 11 16:19:23.845909 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver six.reraise(c, e, tb)
Apr 11 16:19:23.846168 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
Apr 11 16:19:23.846445 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver rv = meth(*args, **kwargs)
Apr 11 16:19:23.846707 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 766, in blockJobAbort
Apr 11 16:19:23.846960 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver if ret == -1: raise libvirtError ('virDomainBlockJobAbort() failed', dom=self)
Apr 11 16:19:23.878201 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver libvirtError: internal error: unable to execute QEMU command 'block-job-complete': The active block job 'drive-virtio-disk1' cannot be completed
Apr 11 16:19:23.878506 ubuntu-xenial-inap-mtl01-0003447058 nova-compute[31776]: ERROR nova.virt.libvirt.driver

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20swap%20volume%5C%22%20AND%20message%3A%5C%22libvirtError%3A%20block%20copy%20still%20active%3A%20domain%20has%20active%20block%20job%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

7 hits in 7 days, check and gate, all failures.

Revision history for this message
Lee Yarwood (lyarwood) wrote :

Zero hits, I can't recall how this was resolved but we are not seeing it anymore.

Changed in nova:
status: Confirmed → Incomplete
importance: Medium → Low
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

Changed in nova:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.