Snapshots left in undeletable state

What volume driver do you use? Can you post nova-volume logs related to this bug?

On Thu, Nov 10, 2011 at 05:43:59PM -0000, Ollie Leahy wrote:
> $ euca-create-volume -s 1 -z nova
> VOLUME vol-0000007c 1 creating (bocktest, None, None, None) 2011-11-10T17:36:41Z
> # euca-create-snapshot vol-0000007c
> SNAPSHOT snap-00000018 vol-0000007c creating 2011-11-10T17:37:33Z 0%
> # euca-delete-volume vol-0000007c# euca-delete-volume vol-0000007c
> VOLUME vol-0000007c

This volume in LVM with snapshot shouldn't be deleted
with VolumeIsBusy exception.
(At least it's my intension of VolumeDriver::delete_volume() with lvdisplay)
Can you please track down why it can be deleted?

If this volume in LVM is deleted, the all of derived snapshots are
deleted at the same time. So the following lvremove for snapshot
would fail. It explains the below behavior.

thanks,
--
yamahata

    I'm mostly using our own driver, but for the logs attached I'm using the
    default driver (nova.volume.driver.ISCSIDriver, I believe)

    When I went back and looked at the behavior of the default driver I saw
    that it's slightly different to the behavior I described in the bug report,
    though it's still broken, I think.

    If I create a volume, then create a snapshot from the volume, and then try
    to delete the base volume, nova-volumes attempts to delete the volume, but
    fails and the volume is left in the stat 'error-deleting'

    In fact I can subsequently delete the snapsnot, but I can't delete the
    volume.

# euca-create-volume -s 1 -z nova
VOLUME vol-0000009d 1 creating (bocktest, None, None, None) 2011-11-15T12:53:25Z
# euca-create-snapshot vol-0000009d
SNAPSHOT snap-00000022 vol-0000009d creating 2011-11-15T12:53:46Z 0%
# euca-delete-volume vol-0000009d
VOLUME vol-0000009d

# euca-describe-volumes
VOLUME vol-0000009d 1 nova error_deleting (bocktest, #########, None, None) 2011-11-15T12:53:25Z

# euca-delete-volume vol-0000009d
ApiError: Volume status must be available

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Yuriy Taraday
Sent: 14 November 2011 19:05
To: Leahy, Oliver
Subject: [Bug 888649] Re: Snapshots left in undeletable state

What volume driver do you use? Can you post nova-volume logs related to
this bug?

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/888649

Title:
  Snapshots left in undeletable state

Status in OpenStack Compute (Nova):
  New

Bug description:
  If a volume is created using the euca api,
                       then a snapshot is created from the volume
                       then the volume is deleted
  the snapshot cannot now be deleted. If the user tries
  to delete the snapshot it ends up in the state 'error_deleting'
  and remains in the system.

  The following sequence of euca commands illustrates the problem

  $ euca-create-volume -s 1 -z nova
  VOLUME vol-0000007c 1 creating (bocktest, None, None, None) 2011-11-10T17:36:41Z
  # euca-create-snapshot vol-0000007c
  SNAPSHOT snap-00000018 vol-0000007c creating 2011-11-10T17:37:33Z 0%
  # euca-delete-volume vol-0000007c# euca-delete-volume vol-0000007c
  VOLUME vol-0000007c
  # euca-delete-snapshot snap-00000018
  SNAPSHOT snap-00000018
  # euca-describe-snapshots
  SNAPSHOT snap-00000018 vol-0000007c error_deleting 2011-11-10T17:37:33Z 100%
  # euca-delete-snapshot snap-00000018
  Traceback (most recent call last):
    File "/usr/bin/euca-delete-snapshot", line 110, in <module>
      main()
    File "/usr/bin/euca-delete-snapshot", line 101, in main
      return_code = euca_conn.delete_snapshot(snapshot_id)
    File "/usr/lib/pymodules/python2.7/boto/ec2/connection.py", line 1112, in delete_snapshot
      return self.get_status('DeleteSnapshot', params)
    File "/usr/lib/pymodules/python2.7/boto/connection.py", line 648, in get_status
      raise self.ResponseError(response...

On my installation I can delete a volume that is referenced by a
snapshot. I will try to understand why this is so and get back to
you.

Though as I said in my previous message the behavior of the LVM based
driver is different to the behavior I described in my original bug
report.

Regards,
Ollie

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Isaku Yamahata
Sent: 14 November 2011 19:06
To: Leahy, Oliver
Subject: Re: [Bug 888649] [NEW] Snapshots left in undeletable state

On Thu, Nov 10, 2011 at 05:43:59PM -0000, Ollie Leahy wrote:
> $ euca-create-volume -s 1 -z nova
> VOLUME vol-0000007c 1 creating (bocktest, None, None, None) 2011-11-10T17:36:41Z
> # euca-create-snapshot vol-0000007c
> SNAPSHOT snap-00000018 vol-0000007c creating 2011-11-10T17:37:33Z 0%
> # euca-delete-volume vol-0000007c# euca-delete-volume vol-0000007c
> VOLUME vol-0000007c

This volume in LVM with snapshot shouldn't be deleted
with VolumeIsBusy exception.
(At least it's my intension of VolumeDriver::delete_volume() with lvdisplay)
Can you please track down why it can be deleted?

If this volume in LVM is deleted, the all of derived snapshots are
deleted at the same time. So the following lvremove for snapshot
would fail. It explains the below behavior.

thanks,
--
yamahata

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/888649

Title:
  Snapshots left in undeletable state

Status in OpenStack Compute (Nova):
  New

Bug description:
  If a volume is created using the euca api,
                       then a snapshot is created from the volume
                       then the volume is deleted
  the snapshot cannot now be deleted. If the user tries
  to delete the snapshot it ends up in the state 'error_deleting'
  and remains in the system.

  The following sequence of euca commands illustrates the problem

  $ euca-create-volume -s 1 -z nova
  VOLUME vol-0000007c 1 creating (bocktest, None, None, None) 2011-11-10T17:36:41Z
  # euca-create-snapshot vol-0000007c
  SNAPSHOT snap-00000018 vol-0000007c creating 2011-11-10T17:37:33Z 0%
  # euca-delete-volume vol-0000007c# euca-delete-volume vol-0000007c
  VOLUME vol-0000007c
  # euca-delete-snapshot snap-00000018
  SNAPSHOT snap-00000018
  # euca-describe-snapshots
  SNAPSHOT snap-00000018 vol-0000007c error_deleting 2011-11-10T17:37:33Z 100%
  # euca-delete-snapshot snap-00000018
  Traceback (most recent call last):
    File "/usr/bin/euca-delete-snapshot", line 110, in <module>
      main()
    File "/usr/bin/euca-delete-snapshot", line 101, in main
      return_code = euca_conn.delete_snapshot(snapshot_id)
    File "/usr/lib/pymodules/python2.7/boto/ec2/connection.py", line 1112, in delete_snapshot
      return self.get_status('DeleteSnapshot', params)
    File "/usr/lib/pymodules/python2.7/boto/connection.py", line 648, in get_status
      raise self.ResponseError(response.status, response.reason, body)
  boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
  <?xml version="1.0"?>
  <Response><Errors><Er...

   Update to describe exact behavior for LVM based driver

    I'm mostly using our own driver in these tests, but I went through the test again using the
    default driver (nova.volume.driver.ISCSIDriver, I believe)

    The behavior of the default driver is slightly different to the behavior I described in the
    original bug report, though it's still broken, I think.

    If I create a volume, then create a snapshot from the volume, and then try
    to delete the base volume, nova-volumes attempts to delete the volume, but
    fails and the volume is left in the stat 'error-deleting'

    In fact I can subsequently delete the snapsnot, but I can't delete the
    volume.

# euca-create-volume -s 1 -z nova
VOLUME vol-0000009d 1 creating (bocktest, None, None, None) 2011-11-15T12:53:25Z
# euca-create-snapshot vol-0000009d
SNAPSHOT snap-00000022 vol-0000009d creating 2011-11-15T12:53:46Z 0%
# euca-delete-volume vol-0000009d
VOLUME vol-0000009d

# euca-describe-volumes
VOLUME vol-00000099 1 nova error (bocktest, stratus10, hostname, None) 2011-11-15T11:37:21Z
VOLUME vol-0000009a 1 nova error_deleting (bocktest, hostname, None, None) 2011-11-15T11:50:06Z
VOLUME vol-0000009d 1 nova error_deleting (bocktest, hostname, None, None) 2011-11-15T12:53:25Z

# euca-delete-volume vol-0000009d
ApiError: Volume status must be available

    I've tracked down the source of the problem deleting volumes that are
    referenced by snapshots using nova.volume.ISCSIDriver.

    When the driver detects that the volume is busy then the delete_volume()
    method in the VolumeDriver class in nova/volume/driver.py raises a
    VolumeIsBusy exception with a keyword arg. However, the VolumeIsBusy
    exception is derived from Error in nova/exception.py and neither
    VolumeIsBusy nor Error accept keyword parameters. This means that an
    unexpected keyword exception is raised, which the driver doesn't expect so
    it sets the status of the volume to 'error-deleting' and stops. See the log
    fragment below.

    Deleting volumes works if I change nova/exception.py so that the
    exception VolumeIsBusy derives from NovaException instead of Error.
    NovaException can handle keyword arguments, so VolumeDrive.delete_volume()
    sees the exception is expects (VolumeIsBusy) and it can handle the
    situation correctly.

    Any opinions on whether this is the 'correct' way to fix this problem?

    regards,
    Ollie

2011-11-16 11:06:46,211 DEBUG nova.utils [-] Running cmd (subprocess): sudo lvdisplay --noheading -C -o Attr nova-volumes/volume-000000a4 from (pid=6234) execute /usr/lib/python2.7/dist-packages/nova/utils.py:165
2011-11-16 11:06:46,271 ERROR nova.volume.manager [-] driver raised exception <__init__() got an unexpected keyword argument 'volume_name'>
2011-11-16 11:06:46,296 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 189, in delete_volume
(nova.rpc): TRACE: raise
(nova.rpc): TRACE: TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType
(nova.rpc): TRACE:

# diff -c /usr/share/pyshared/nova/exception.py{,.backup}
*** /usr/share/pyshared/nova/exception.py 2011-11-16 11:46:31.000000000 +0000
--- /usr/share/pyshared/nova/exception.py.backup 2011-11-16 11:39:04.000000000 +0000
***************
*** 370,376 ****
      message = _("Snapshot %(snapshot_id)s could not be found.")

! class VolumeIsBusy(NovaException):
      message = _("deleting volume %(volume_name)s that has snapshot")

--- 370,376 ----
      message = _("Snapshot %(snapshot_id)s could not be found.")

! class VolumeIsBusy(Error):
      message = _("deleting volume %(volume_name)s that has snapshot")

-----Original Message-----
From: Leahy, Oliver
Sent: 15 November 2011 14:35
To: 'Bug 888649'
Subject: RE: [Bug 888649] [NEW] Snapshots left in undeletable state

On my installation I can delete a volume that is referenced by a
snapshot. I will try to understand why this is so and get back to
you.

Though as I said in my previous message the behavior of the LVM based
driver is different to the behavior I described in my origi...

Things shouldn't be derived from Error, that is for sure.

Can you verify that it actually works as expected with the change?

Vish

On Nov 16, 2011, at 6:46 AM, Ollie Leahy wrote:

> I've tracked down the source of the problem deleting volumes that are
> referenced by snapshots using nova.volume.ISCSIDriver.
>
> When the driver detects that the volume is busy then the delete_volume()
> method in the VolumeDriver class in nova/volume/driver.py raises a
> VolumeIsBusy exception with a keyword arg. However, the VolumeIsBusy
> exception is derived from Error in nova/exception.py and neither
> VolumeIsBusy nor Error accept keyword parameters. This means that an
> unexpected keyword exception is raised, which the driver doesn't expect so
> it sets the status of the volume to 'error-deleting' and stops. See the log
> fragment below.
>
>
> Deleting volumes works if I change nova/exception.py so that the
> exception VolumeIsBusy derives from NovaException instead of Error.
> NovaException can handle keyword arguments, so VolumeDrive.delete_volume()
> sees the exception is expects (VolumeIsBusy) and it can handle the
> situation correctly.
>
> Any opinions on whether this is the 'correct' way to fix this
> problem?
>
> regards,
> Ollie
>
>
>
>
>
> 2011-11-16 11:06:46,211 DEBUG nova.utils [-] Running cmd (subprocess): sudo lvdisplay --noheading -C -o Attr nova-volumes/volume-000000a4 from (pid=6234) execute /usr/lib/python2.7/dist-packages/nova/utils.py:165
> 2011-11-16 11:06:46,271 ERROR nova.volume.manager [-] driver raised exception <__init__() got an unexpected keyword argument 'volume_name'>
> 2011-11-16 11:06:46,296 ERROR nova.rpc [-] Exception during message handling
> (nova.rpc): TRACE: Traceback (most recent call last):
> (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
> (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
> (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/volume/manager.py", line 189, in delete_volume
> (nova.rpc): TRACE: raise
> (nova.rpc): TRACE: TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType
> (nova.rpc): TRACE:
>
>
> # diff -c /usr/share/pyshared/nova/exception.py{,.backup}
> *** /usr/share/pyshared/nova/exception.py 2011-11-16 11:46:31.000000000 +0000
> --- /usr/share/pyshared/nova/exception.py.backup 2011-11-16 11:39:04.000000000 +0000
> ***************
> *** 370,376 ****
> message = _("Snapshot %(snapshot_id)s could not be found.")
>
>
> ! class VolumeIsBusy(NovaException):
> message = _("deleting volume %(volume_name)s that has snapshot")
>
>
> --- 370,376 ----
> message = _("Snapshot %(snapshot_id)s could not be found.")
>
>
> ! class VolumeIsBusy(Error):
> message = _("deleting volume %(volume_name)s that has snapshot")
>
>
>
> -----Original Message-----
> From: Leahy, Oliver
> Sent: 15 November 2011 14:35
> To: 'Bug 888649'
> Subject: RE: [Bug 888649] [NEW] Snapshots left in undeletable state
>
> On my installation I can delete a volume that is referenced by a
> ...

Yes, when I make that change the operations I'm doing work as expected.
That is, when I try to delete a volume that is referenced by a snapshot,
the volume driver detects that the volume is busy and leaves it in the
'available' state. I can continue to use the volume and the snapshot,
and I can subsequently delete the snapshot and then the volume.

The only observation I have is that euca-delete-volume returns success
when I try to delete the volume referenced by a snapshot. We think this
is acceptable.

In fact, I think that it's not possible to pass an error status back
from the nova-volume driver, through nova-api to the cli. So the only
way an error status could be returned to the CLI is if nova-api detected
the busy condition and didn't sent the request to nova-volume in the
case where the volume is being referenced.

But, we would not like nova-api to stop the delete request being sent to
the driver. Because we are developing a nova-volume driver that will
support the deleting volumes that are referenced by snapshots. We see
valid use cases for this and expect that other development groups would
too.

Reviewed: https://review.openstack.org/1900
Committed: http://github.com/openstack/nova/commit/e0ef89f091a77a25fa9bcfd466159e101cb42c56
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit e0ef89f091a77a25fa9bcfd466159e101cb42c56
Author: Ollie Leahy <email address hidden>
Date: Thu Nov 24 15:56:08 2011 +0000

    Fixes bug 888649

    Change exception.VolumeIsBusy to derive from NovaException instead of
    Error, so that an 'unexpected keyword' exception is not thrown when
    the exception is raised with keyword parameters.

    Add unit test to confirm that the 'unexpected keyword' exception is not
    thrown when the exception is raised by nova.volume.driver.VolumeDriver

    Responded to reviewer observations, fix pep8 errors in tset_volume.py,
    added email address to Authors file.

    Change-Id: I15464cb0cf72a2c71f430e4c8c5c2b27cd4e2ef9

Reviewed: https://review.openstack.org/2185
Committed: http://github.com/openstack/nova/commit/5d319531552689e4842a84adce01c81899cdfb17
Submitter: Jenkins
Branch: stable/diablo

commit 5d319531552689e4842a84adce01c81899cdfb17
Author: Ollie Leahy <email address hidden>
Date: Thu Nov 24 15:56:08 2011 +0000

    Fixes bug 888649

    Change exception.VolumeIsBusy to derive from NovaException instead of
    Error, so that an 'unexpected keyword' exception is not thrown when
    the exception is raised with keyword parameters.

    Add unit test to confirm that the 'unexpected keyword' exception is not
    thrown when the exception is raised by nova.volume.driver.VolumeDriver

    Responded to reviewer observations, fix pep8 errors in tset_volume.py,
    added email address to Authors file.

    (cherry picked from commit e0ef89f091a77a25fa9bcfd466159e101cb42c56)

    Change-Id: I15464cb0cf72a2c71f430e4c8c5c2b27cd4e2ef9