OpenStack Compute (nova)

libvirt.libvirtError: unable to verify existence of block copy target: Permission denied

Bug #1885418 reported by Maysam Fazeli on 2020-06-28

This bug report is a duplicate of: Bug #1896617: [SRU] Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri. Edit Remove

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Incomplete	Undecided	Unassigned

Bug Description

Creating a live snapshot of local root volume storage fails with the following error:

2020-06-28 06:21:03.673 167084 INFO nova.compute.manager [req-1af43725-b524-47ce-8ed6-d3a9dffdc65c 220423f8f62241b59a4f7cf5ee2e41a2 b2e1e496515948598349f8ea7ec7c2c9 - ba00b19e5674426caf6ea12d5fecfad1 ba00b19e5674426caf6ea12d5fecfad1] [instance: 17be6008-70f9-4669-bfd5-e1017b2467f3] instance snapshotting
2020-06-28 06:21:03.771 167084 INFO nova.virt.libvirt.driver [req-1af43725-b524-47ce-8ed6-d3a9dffdc65c 220423f8f62241b59a4f7cf5ee2e41a2 b2e1e496515948598349f8ea7ec7c2c9 - ba00b19e5674426caf6ea12d5fecfad1 ba00b19e5674426caf6ea12d5fecfad1] [instance: 17be6008-70f9-4669-bfd5-e1017b2467f3] Beginning live snapshot process
2020-06-28 06:21:04.053 167084 INFO nova.virt.libvirt.driver [req-1af43725-b524-47ce-8ed6-d3a9dffdc65c 220423f8f62241b59a4f7cf5ee2e41a2 b2e1e496515948598349f8ea7ec7c2c9 - ba00b19e5674426caf6ea12d5fecfad1 ba00b19e5674426caf6ea12d5fecfad1] [instance: 17be6008-70f9-4669-bfd5-e1017b2467f3] Skipping quiescing instance: QEMU guest agent is not enabled.
2020-06-28 06:21:04.256 167084 INFO nova.compute.manager [req-1af43725-b524-47ce-8ed6-d3a9dffdc65c 220423f8f62241b59a4f7cf5ee2e41a2 b2e1e496515948598349f8ea7ec7c2c9 - ba00b19e5674426caf6ea12d5fecfad1 ba00b19e5674426caf6ea12d5fecfad1] [instance: 17be6008-70f9-4669-bfd5-e1017b2467f3] Successfully reverted task state from image_pending_upload on failure for instance.
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server [req-1af43725-b524-47ce-8ed6-d3a9dffdc65c 220423f8f62241b59a4f7cf5ee2e41a2 b2e1e496515948598349f8ea7ec7c2c9 - ba00b19e5674426caf6ea12d5fecfad1 ba00b19e5674426caf6ea12d5fecfad1] Exception during message handling: libvirt.libvirtError: unable to verify existence of block copy target: Permission denied
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 2351, in snapshot
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server metadata['location'] = root_disk.direct_snapshot(
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 452, in direct_snapshot
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server raise NotImplementedError(_('direct_snapshot() is not implemented'))
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not implemented
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 77, in wrapped
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server _emit_exception_notification(
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server raise value
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 188, in decorated_function
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. "
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server raise value
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 159, in decorated_function
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1447, in decorated_function
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 216, in decorated_function
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server raise value
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 205, in decorated_function
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3844, in backup_instance
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server self._do_snapshot_instance(context, image_id, instance)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 236, in decorated_function
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server compute_utils.delete_image(
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server raise value
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 232, in decorated_function
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server return function(self, context, image_id, instance,
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3830, in _do_snapshot_instance
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server self._snapshot_instance(context, image_id, instance,
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 3912, in _snapshot_instance
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server self.driver.snapshot(context, instance, image_id,
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 2390, in snapshot
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server self._live_snapshot(context, instance, guest,
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 2599, in _live_snapshot
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server dev.rebase(disk_delta, copy=True, reuse_ext=True, shallow=True)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 796, in rebase
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server return self._guest._domain.blockRebase(
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server raise value
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/libvirt.py", line 1079, in blockRebase
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server if ret == -1: raise libvirtError ('virDomainBlockRebase() failed', dom=self)
2020-06-28 06:21:04.259 167084 ERROR oslo_messaging.rpc.server libvirt.libvirtError: unable to verify existence of block copy target: Permission denied

Latest OpenStack Ussuri on Ubuntu 20.04 + libvirtd 6.0.0 and Python 3.8.2

Tags:

Balazs Gibizer (balazs-gibizer) on 2020-08-06

tags:

added: libvirt snapshot

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2020-08-06:

Download full text (3.1 KiB)

Would you be able to update the description using the reporting guidelines listed below?

Specifically I'd like to understand which volume backend is being used, what commands were used etc.

OpenStack Compute (nova) bug reporting guidelines:
!!!!!!!!!!!!!!!!!!!!!!!!! READ THIS !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Each bug report needs to provide a minimum of information without
we are not able to address the issue you observed. It is crucial
for other developers to have this information. You can use the
template below, which asks for this information.

"Request for Feature Enhancements" (RFEs) are collected:
* with the blueprint process [1][2][3] (if you're a developer) OR
* via the liason with the operator (ops) community [4].
This means "wishlist" bugs won't get any attention anymore.

You can ask in the #openstack-nova IRC channel on freenode, if you have questions about this.

References:
[1] https://blueprints.launchpad.net/nova/
[2] https://github.com/openstack/nova-specs
[3] https://wiki.openstack.org/wiki/Blueprints
[4] http://lists.openstack.org/pipermail/openstack-operators/2016-March/010007.html

!!!!!!!!!!!!!!!!!!!!!!!!! READ THIS !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Description
===========
Some prose which explains more in detail what this bug report is
about. If the headline of this report is descriptive enough, skip
this section.

Steps to reproduce
==================
A chronological list of steps which will bring off the
issue you noticed:
* I did X
* then I did Y
* then I did Z
A list of openstack client commands (with correct argument value)
would be the most descriptive example. To get more information use:

    $ nova --debug <command> <arg1> <arg2=value>
    or
    $ openstack --debug <command> <arg1> <arg2=value>

Expected result
===============
After the execution of the steps above, what should have
happened if the issue wasn't present?

Actual result
=============
What happened instead of the expected result?
How did the issue look like?

Environment
===========
1. Exact version of OpenStack you are running. See the following
list for all releases: http://docs.openstack.org/releases/

   If this is from a distro please provide
       $ dpkg -l | grep nova
       or
       $ rpm -ql | grep nova
   If this is from git, please provide
       $ git log -1

2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?

2. Which storage type did you use?
(For example: Ceph, LVM, GPFS, ...)
What's the version of that?

3. Which networking type did you use?
(For example: nova-network, Neutron with OpenVSwitch, ...)

Logs & Configs
==============
The tool *sosreport* has support for some OpenStack projects.
It's worth having a look at it. For example, if you want to collect
the logs of a compute node you would execute:

$ sudo sosreport -o openstack_nova --batch

on that compute node. Attach the logs to this bug report. Please
consider that these logs need to be collected in "DEBUG" mode.

For tips on reporting VMware virt driver bugs, please see this doc: https://docs.openstack.org/nova/latest/admin/configuration/hypervisor-vmware.html#troubleshooti...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Changed in nova:
status:	New → Incomplete