Cannot delete images when rbd store is used

I'm guessing this is the same issue as https://bugs.launchpad.net/glance/+bug/1075580. Can you try using the patch merged there?

If that patch doesn't solve it, could you apply this one to get more information about what the actual error is:

diff --git a/glance/api/v1/images.py b/glance/api/v1/images.py
index 55d7cc2..219bb89 100644
--- a/glance/api/v1/images.py
+++ b/glance/api/v1/images.py
@@ -830,7 +830,9 @@ class Controller(controller.BaseController):
             # Delete the image from the registry first, since we rely on it
             # for authorization checks.
             # See https://bugs.launchpad.net/glance/+bug/1065187
+ LOG.debug('updating image status in registry')
             registry.update_image_metadata(req.context, id, {'status': status})
+ LOG.debug('deleting image metadata from registry')
             registry.delete_image_metadata(req.context, id)

             # The image's location field may be None in the case
@@ -841,8 +843,10 @@ class Controller(controller.BaseController):
                 if CONF.delayed_delete:

schedule_delayed_delete_from_backend(image['location'], id)
                 else:
+ LOG.debug('deleting image from backend store')
                     safe_delete_from_backend(image['location'],
                                              req.context, id)
+ LOG.debug('finished deleting image')
         except exception.NotFound, e:
             msg = ("Failed to find image to delete: %(e)s" % locals())
             for line in msg.split('\n'):
@@ -859,6 +863,8 @@ class Controller(controller.BaseController):
             raise HTTPForbidden(explanation=msg,
                                 request=req,
                                 content_type="text/plain")
+ except:
+ LOG.exception('error deleting image through v1 api')
         else:
             self.notifier.info('image.delete', image)

diff --git a/glance/store/rbd.py b/glance/store/rbd.py
index dbc693a..337b371 100644
--- a/glance/store/rbd.py
+++ b/glance/store/rbd.py
@@ -276,15 +276,19 @@ class Store(glance.store.base.Store):
                 if loc.snapshot:
                     with rbd.Image(ioctx, loc.image) as image:
                         try:
+ LOG.debug('unprotecting snapshot')
                             image.unprotect_snap(loc.snapshot)
                         except rbd.ImageBusy:
                             log_msg = _("snapshot %s@%s could not be "
                                         "unprotected because it is in use")
                             LOG.error(log_msg % (loc.image, loc.snapshot))
                             raise exception.InUseByStore()
+ LOG.debug('removing snapshot')
                         image.remove_snap(loc.snapshot)
                 try:
+ LOG.debug('removing rbd image')
                     rbd.RBD().remove(ioctx, str(loc.image))
+ LOG.debug('successfully removed rbd image')
                 except rbd.ImageNotFound:
                     raise exception.NotFound(
                         _('RBD image %s does not exist') % loc.image)

Steve, have you had a chance to try Josh's suggestion?

Sorry for my late response on this, have only recently had a change to look at this again.

I've upgraded to the latest versions of glance, ceph and Nova as follows:
glance - glance-api (= 2012.2-0ubuntu2.3~cloud0), glance-registry (= 2012.2-0ubuntu2.3~cloud0)
Nova - 2012.2-0ubuntu5~cloud0
python-ceph - Version: 0.56.1-1precise

the behavior has changed slightly since the upgrades were done:

using glance v1.0 api

glance index
ID Name Disk Format Container Format Size
------------------------------------ ------------------------------ -------------------- -------------------- --------------
0a6046d6-18dc-43b2-9155-11a46a456735 sles11 qcow2 ovf 2952331264
18be5c40-217f-41a6-b899-c988ab24ba67 winabcdef qcow2 ovf 11302207488

glance delete 0a6046d6-18dc-43b2-9155-11a46a456735
Delete image 0a6046d6-18dc-43b2-9155-11a46a456735? [y/N] y
Request returned failure status.
None
HTTPInternalServerError (HTTP 500)

500 error is returned again

glance index
ID Name Disk Format Container Format Size
------------------------------------ ------------------------------ -------------------- -------------------- --------------
18be5c40-217f-41a6-b899-c988ab24ba67 winabcdef qcow2 ovf 11302207488

however the image does now delete

using v2 api

 glance image-list
+--------------------------------------+-----------------------+-------------+------------------+-------------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+-----------------------+-------------+------------------+-------------+--------+
| 18be5c40-217f-41a6-b899-c988ab24ba67 | winabcdef | qcow2 | ovf | 11302207488 | active |
+--------------------------------------+-----------------------+-------------+------------------+-------------+--------+

glance image-delete 18be5c40-217f-41a6-b899-c988ab24ba67
Request returned failure status.
None
HTTPInternalServerError (HTTP 500)

again, the 500 error is returned

root@iswubu33:/var/lib/glance/temp# glance image-list
+--------------------------------------+-----------------------+-------------+------------------+-------------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+-----------------------+-------------+------------------+-------------+--------+
|
+--------------------------------------+-----------------------+-------------+------------------+-------------+--------+

the image however is deleted.

so the actual delete problem is resolved, but there is still an error returned from the client - this is the case via the CLI or via Horizon.

I then uploaded a new image - UUID reported by glance as follows for a v1 delete operation - the UUID I deleted was 2e7468d9-97...

Hi Steve, thanks for the update. I think the change in behavior (it's marked as deleted in the registry now, but still gives a 500 error) is due to a change in glance to make it update the registry first before deleting from the backend store, so the original problem just happens later.

Do you have cephx authentication enabled? If so, what caps (as shown in 'ceph auth list') does client.glance (or whatever user you're using with glance) have?

The caps can be set to correct values (as shown in http://ceph.com/docs/master/rbd/rbd-openstack/#setup-ceph-client-authentication) with:

ceph auth get-or-create client.tests mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'

If you're not using cephx, or that doesn't fix the problem, the full files with extra debugging are here:

glance/api/v1/images.py:
http://paste.openstack.org/raw/29935/

glance/store/rbd.py:
http://paste.openstack.org/raw/29936/

Just the glance api log with the extra debugging is needed.

I'm very new to openstack and ceph (first installation) and I'm seeing a simlar error. The output from

glance --debug image-delete d938b672-af7c-4950-ad24-2a571f66bc4e
curl -i -X DELETE -H 'X-Auth-Token: 48e7b767956c4c6aaa11c2f9b42e3e12' -H 'Content-Type: application/octet-stream' -H 'User-Agent: python-glanceclient' http://192.168.137.69:9292/v1/images/d938b672-af7c-4950-ad24-2a571f66bc4e

HTTP/1.1 500 Internal Server Error
date: Fri, 08 Feb 2013 18:11:38 GMT
content-length: 3537
content-type: text/plain
connection: close

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 382, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 326, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 278, in __call__
    return self.app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 326, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 532, in __call__
    request, **action_args)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 549, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python2.7/d...

In your case the problem is clearly with the ceph capabilities given to the clients. Which version of ceph are you running?
There was a bug in 0.55 that would cause that problem to occur with those caps.

I'm runnng ceph version 0.56.2-1quantal.

I can't reproduce the problem locally.
Could you double-check (with ceph-osd -v) that every osd is running 0.56.2?

Another possibility is some extraneous character in the caps for client.images.
You can test by using e.g. 'rados --id images -k /patch/to/keyring -p rbd lock list rbd_children'.
If the caps are right, it should say 'No such file or directory' (assuming the rbd_children object doesn't exist in that pool).
Given the error from glance, I'm guessing it'll say 'Operation not permitted' right now.

Stephen Street:

Someone else had a similar problem [1], and the issue ended up being new behavior in boost::spirit, specifically the version in quantal.

The fix will be in bobtail 0.56.4, but in the meantime you can run the bobtail branch (http://ceph.com/docs/master/install/debian/#development-testing-packages) to get it.

The original issue in this bug is still unknown, since it occurred on precise.

[1] http://tracker.ceph.com/issues/4122

Josh - is there anything we need to do in Glance?

Brian:
I'm pretty sure this is a ceph or configuration issue if it's still happening. You can close this glance bug if you like, and ask that further updates go to the ceph bug tracker (http://tracker.ceph.com/projects/rbd/issues/new).