OpenStack Shared File Systems Service (Manila)

LVM driver fails to delete already deleted shares

Bug #1888915 reported by Goutham Pacha Ravi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Undecided
Goutham Pacha Ravi

Bug Description

Description
===========
This issue is seen in the CI more often than in real deployments, although it's possible there as well. The LVM driver may have failed creation of a share, share snapshot or a group snapshot, and the user may want to delete the failed resource. The recommended way for share drivers to handle deletion of already deleted resources is to log a warning and return to the share manager, and not fail the operation. Failing the operation will result in the resource's status being set to "error" or "error_deleting" - a state that requires administrator intervention to investigate and correct.

This applies to shares and snapshots. To reliably reproduce, you could:
1) create a manila share on the LVM driver
2) create a snapshot
3) delete the snapshot externally with "lvremove -f"
4) delete the snapshot via manila, check the manila-share log, the error is:

     ERROR oslo_messaging.rpc.server Traceback (most recent call last):
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
    ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 273, in dispatch
    ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 193, in _do_dispatch
    ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 568, in wrapper
    ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 4368, in delete_share_group_snapshot
    ERROR oslo_messaging.rpc.server snap_ref['name'])
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    ERROR oslo_messaging.rpc.server self.force_reraise()
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
    ERROR oslo_messaging.rpc.server raise value
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 4349, in delete_share_group_snapshot
    ERROR oslo_messaging.rpc.server context, snap_ref, share_server=share_server))
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/driver.py", line 1707, in delete_share_group_snapshot
    ERROR oslo_messaging.rpc.server context, share_snapshot, share_server=share_server)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/lvm.py", line 435, in delete_snapshot
    ERROR oslo_messaging.rpc.server share_server)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/lvm.py", line 146, in delete_snapshot
    ERROR oslo_messaging.rpc.server self._deallocate_container(snapshot['name'])
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/lvm.py", line 109, in _deallocate_container
    ERROR oslo_messaging.rpc.server share_name), run_as_root=True)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/driver.py", line 221, in _try_execute
    ERROR oslo_messaging.rpc.server self._execute(*command, **kwargs)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 101, in execute
    ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py", line 434, in execute
    ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
    ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
    ERROR oslo_messaging.rpc.server Command: sudo manila-rootwrap /etc/manila/rootwrap.conf lvremove -f lvm-shares/share-snapshot-fb495959-052f-4c55-b901-3b3ab93089dd
    ERROR oslo_messaging.rpc.server Exit code: 5
    ERROR oslo_messaging.rpc.server Stdout: ''
    ERROR oslo_messaging.rpc.server Stderr: ' Failed to find logical volume "lvm-shares/share-snapshot-fb495959-052f-4c55-b901-3b3ab93089dd"\n'

5) force delete the snapshot
6) unmount the lvm share and delete the share externally
7) try deleting the manila share, check manila log:

    Command: sudo manila-rootwrap /etc/manila/rootwrap.conf umount -f /opt/stack/data/manila/mnt/share-d9f2c67e-e664-4116-bb90-c4732a54965b
    Exit code: 32
    Stdout: ''
    Stderr: 'umount: /opt/stack/data/manila/mnt/share-d9f2c67e-e664-4116-bb90-c4732a54965b: not mounted.\n'
    ERROR oslo_messaging.rpc.server Traceback (most recent call last):
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
    ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 273, in dispatch
    ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 193, in _do_dispatch
    ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 187, in wrapped
    ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 568, in wrapper
    ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 3095, in delete_share_instance
    ERROR oslo_messaging.rpc.server exception=excep)
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    ERROR oslo_messaging.rpc.server self.force_reraise()
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
    ERROR oslo_messaging.rpc.server raise value
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 3070, in delete_share_instance
    ERROR oslo_messaging.rpc.server share_server=share_server)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/lvm.py", line 259, in delete_share
    ERROR oslo_messaging.rpc.server self._unmount_device(share)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/lvm.py", line 269, in _unmount_device
    ERROR oslo_messaging.rpc.server self._execute('umount', '-f', mount_path, run_as_root=True)
    ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 101, in execute
    ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
    ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_concurrency/processutils.py", line 434, in execute
    ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
    ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
    ERROR oslo_messaging.rpc.server Command: sudo manila-rootwrap /etc/manila/rootwrap.conf umount -f /opt/stack/data/manila/mnt/share-d9f2c67e-e664-4116-bb90-c4732a54965b
    ERROR oslo_messaging.rpc.server Exit code: 32
    ERROR oslo_messaging.rpc.server Stdout: ''
    ERROR oslo_messaging.rpc.server Stderr: 'umount: /opt/stack/data/manila/mnt/share-d9f2c67e-e664-4116-bb90-c4732a54965b: not mounted.\n'
    ERROR oslo_messaging.rpc.server

This code path needs hardening.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/743438

Changed in manila:
assignee: nobody → Goutham Pacha Ravi (gouthamr)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.opendev.org/743438
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=e57809fd74186714aec7dae305380a51c31fc7d5
Submitter: Zuul
Branch: master

commit e57809fd74186714aec7dae305380a51c31fc7d5
Author: Goutham Pacha Ravi <email address hidden>
Date: Mon Jul 27 18:02:51 2020 -0700

    Harden LVM driver deletion paths

    During maintenance, administrators may decide
    to unmount shares, snapshots or remove them
    entirely prior to cleaning up LVM share resources
    in manila. The driver should not fail on deletion
    of missing resources.

    Change-Id: Ieaf37ec10db9a8bdce6bb195b76335fea9b2b52f
    Closes-Bug: #1888915
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/743642

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/ussuri)

Reviewed: https://review.opendev.org/743642
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=8620f7efed1c2b669a06f13440add1eb20ef12e9
Submitter: Zuul
Branch: stable/ussuri

commit 8620f7efed1c2b669a06f13440add1eb20ef12e9
Author: Goutham Pacha Ravi <email address hidden>
Date: Mon Jul 27 18:02:51 2020 -0700

    Harden LVM driver deletion paths

    During maintenance, administrators may decide
    to unmount shares, snapshots or remove them
    entirely prior to cleaning up LVM share resources
    in manila. The driver should not fail on deletion
    of missing resources.

    Change-Id: Ieaf37ec10db9a8bdce6bb195b76335fea9b2b52f
    Closes-Bug: #1888915
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit e57809fd74186714aec7dae305380a51c31fc7d5)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/743743

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/train)

Reviewed: https://review.opendev.org/743743
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=4e292ea884ff72341c2ab6733745fe6d35733aa8
Submitter: Zuul
Branch: stable/train

commit 4e292ea884ff72341c2ab6733745fe6d35733aa8
Author: Goutham Pacha Ravi <email address hidden>
Date: Mon Jul 27 18:02:51 2020 -0700

    Harden LVM driver deletion paths

    During maintenance, administrators may decide
    to unmount shares, snapshots or remove them
    entirely prior to cleaning up LVM share resources
    in manila. The driver should not fail on deletion
    of missing resources.

    Change-Id: Ieaf37ec10db9a8bdce6bb195b76335fea9b2b52f
    Closes-Bug: #1888915
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit e57809fd74186714aec7dae305380a51c31fc7d5)
    (cherry picked from commit 8620f7efed1c2b669a06f13440add1eb20ef12e9)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/743840

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/743854

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/stein)

Reviewed: https://review.opendev.org/743840
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=c441ec22efc8a0d1579497289bed9ed0c9f8c8c0
Submitter: Zuul
Branch: stable/stein

commit c441ec22efc8a0d1579497289bed9ed0c9f8c8c0
Author: Goutham Pacha Ravi <email address hidden>
Date: Mon Jul 27 18:02:51 2020 -0700

    Harden LVM driver deletion paths

    During maintenance, administrators may decide
    to unmount shares, snapshots or remove them
    entirely prior to cleaning up LVM share resources
    in manila. The driver should not fail on deletion
    of missing resources.

    Change-Id: Ieaf37ec10db9a8bdce6bb195b76335fea9b2b52f
    Closes-Bug: #1888915
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit e57809fd74186714aec7dae305380a51c31fc7d5)
    (cherry picked from commit 8620f7efed1c2b669a06f13440add1eb20ef12e9)
    (cherry picked from commit 4e292ea884ff72341c2ab6733745fe6d35733aa8)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/rocky)

Reviewed: https://review.opendev.org/743854
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=40abc1bcf676727c2f36bb523afcb4d383332f10
Submitter: Zuul
Branch: stable/rocky

commit 40abc1bcf676727c2f36bb523afcb4d383332f10
Author: Goutham Pacha Ravi <email address hidden>
Date: Mon Jul 27 18:02:51 2020 -0700

    Harden LVM driver deletion paths

    During maintenance, administrators may decide
    to unmount shares, snapshots or remove them
    entirely prior to cleaning up LVM share resources
    in manila. The driver should not fail on deletion
    of missing resources.

    Change-Id: Ieaf37ec10db9a8bdce6bb195b76335fea9b2b52f
    Closes-Bug: #1888915
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit e57809fd74186714aec7dae305380a51c31fc7d5)
    (cherry picked from commit 8620f7efed1c2b669a06f13440add1eb20ef12e9)
    (cherry picked from commit 4e292ea884ff72341c2ab6733745fe6d35733aa8)
    (cherry picked from commit c441ec22efc8a0d1579497289bed9ed0c9f8c8c0)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/744043

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/queens)

Reviewed: https://review.opendev.org/744043
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=cdf1b7b17d361f857b47dc06d561c1fa0de86857
Submitter: Zuul
Branch: stable/queens

commit cdf1b7b17d361f857b47dc06d561c1fa0de86857
Author: Goutham Pacha Ravi <email address hidden>
Date: Mon Jul 27 18:02:51 2020 -0700

    Harden LVM driver deletion paths

    During maintenance, administrators may decide
    to unmount shares, snapshots or remove them
    entirely prior to cleaning up LVM share resources
    in manila. The driver should not fail on deletion
    of missing resources.

    Change-Id: Ieaf37ec10db9a8bdce6bb195b76335fea9b2b52f
    Closes-Bug: #1888915
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit e57809fd74186714aec7dae305380a51c31fc7d5)
    (cherry picked from commit 8620f7efed1c2b669a06f13440add1eb20ef12e9)
    (cherry picked from commit 4e292ea884ff72341c2ab6733745fe6d35733aa8)
    (cherry picked from commit c441ec22efc8a0d1579497289bed9ed0c9f8c8c0)
    (cherry picked from commit 40abc1bcf676727c2f36bb523afcb4d383332f10)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 10.0.2

This issue was fixed in the openstack/manila 10.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila queens-eol

This issue was fixed in the openstack/manila queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila rocky-eol

This issue was fixed in the openstack/manila rocky-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.