OpenStack Object Storage (swift)

can not delete only metada of container

Bug #1049017 reported by Yukihiro KAWADA on 2012-09-11

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Object Storage (swift)	Fix Released	Undecided	Tom Fifield	OpenStack Object Storage (swift) 1.7.5

Bug Description

We can not delete only metada of container.

Example is case of StaticWeb:
container=web
file=web/index.html

curl -v -X GET -H 'X-Storage-Token: AUTH_tk168fccd695e84f2693b6fc966cb69d14' http://144.50.113.13:8080/v1/AUTH_system/web

< HTTP/1.1 200 OK
< X-Container-Object-Count: 1
< X-Container-Meta-Web-Listing: true
< X-Timestamp: 1347270749.13087
< X-Container-Read: .r:*
< X-Container-Bytes-Used: 3686
< X-Container-Meta-Web-Index: index.html
< Accept-Ranges: bytes
< Content-Length: 11
< Content-Type: text/plain; charset=utf-8
< Date: Tue, 11 Sep 2012 09:15:35 GMT
index.html

Now I want to delete 'X-Container-Meta-Web-Index: index.html' field, but can not.
I have no choice but to delete container 'web'.
I can only delete all of container 'web' and metadata.

My patch is very simple fix.
If I set 'deleteme' to metadata value, then delete its metadata item

curl -v -X PUT -H 'X-Container-Meta-Web-Index: deleteme' -H 'X-Storage-Token: AUTH_tk168fccd695e84f2693b6fc966cb69d14' http://144.50.113.13:8080/v1/AUTH_system/web

< HTTP/1.1 200 OK
< X-Container-Object-Count: 1
< X-Container-Meta-Web-Listing: true
< X-Timestamp: 1347270749.13087
< X-Container-Read: .r:*
< X-Container-Bytes-Used: 3686
< Accept-Ranges: bytes
< Content-Length: 11
< Content-Type: text/plain; charset=utf-8
< Date: Tue, 11 Sep 2012 09:27:41 GMT
index.html

Thank you.

=== here is patch
diff --git a/swift/common/db.py b/swift/common/db.py
index 6b30db4..f6fd972 100644
--- a/swift/common/db.py
+++ b/swift/common/db.py
@@ -588,6 +588,9 @@ class DatabaseBroker(object):
                 value, timestamp = value_timestamp
                 if key not in md or timestamp > md[key][1]:
                     md[key] = value_timestamp
+ ## if remove metadata item then set 'deleteme' to its header value
+ if value == 'deleteme':
+ del md[key]
             conn.execute('UPDATE %s_stat SET metadata = ?' % self.db_type,
                          (json.dumps(md),))
             conn.commit()
===

See original description

Yukihiro KAWADA (warp-kawada) on 2012-09-11

description:

updated

Revision history for this message

Alex Yang (alexyang) wrote on 2012-09-11:

If you want delete a custom metadata, just POST a 'X-Remove-Container-Meta-Web-Index: anything' or 'X-Remove-Container-Meta-Web-Index: '.

The curl can't transfer an empty header, so use the 'X-Remove-Container-Meta-Web-Index: anything'.

Changed in swift:
status:	New → Invalid

Revision history for this message

gholt (gholt) wrote on 2012-09-11:

For future reference: http://docs.openstack.org/api/openstack-object-storage/1.0/content/delete-container-metadata.html

Revision history for this message

Yukihiro KAWADA (warp-kawada) wrote on 2012-09-12:

The curl 7.23.0 or later can send empty headers if you have it.

curl.1:
-H
:
If you send the custom header with no-value then its
header must be terminated with a semicolon, such as -H "X-Cus-
tom-Header;" to send "X-Custom-Header:".

http://permalink.gmane.org/gmane.comp.web.curl.announce/62

Is X-Remove-Container blueprint only?
I could not find its implementations.
And I think this feature needs many costs.
How long should I waiting for it?

Thank you.

Revision history for this message

Alex Yang (alexyang) wrote on 2012-09-12:

The 'X-Remove-*-Meta-*' the already in swift/proxy/controllers/base.py [line 99~108]

Revision history for this message

Yukihiro KAWADA (warp-kawada) wrote on 2012-09-12:

I'm sorry. I could find in 1.6.0 , too.
But does not this work it yet?
I've tried it with 1.6.0. I didn't get to expect.
I think db.py needs to update.

Revision history for this message

Yukihiro KAWADA (warp-kawada) wrote on 2012-09-12:

Oh... should use reclaim function in db.py.
It looks like just same as my codes.
I did not notice.
But it does not work for me, yet.

Revision history for this message

gholt (gholt) wrote on 2012-09-12:

The empty header form of deletion has been there since August of 2010 (version 1.1.0).

The -remove- form of deletion has been there since March of this year (version 1.4.8).

The documentation for such is at http://docs.openstack.org/api/openstack-object-storage/1.0/content/delete-container-metadata.html

Here's a quick test I did with curl:

# Create container
$ curl -XPUT http://127.0.0.1:8080/v1/AUTH_test/test_container -Hx-auth-token:AUTH_tk71b0ad0c9b81435884091ae1ffea0ec1
201 Created

# HEAD container to show no metadata
$ curl -I http://127.0.0.1:8080/v1/AUTH_test/test_container -Hx-auth-token:AUTH_tk71b0ad0c9b81435884091ae1ffea0ec1
HTTP/1.1 204 No Content
X-Container-Object-Count: 0
X-Timestamp: 1347421947.14939
X-Container-Bytes-Used: 0
Accept-Ranges: bytes
Content-Length: 0
Date: Wed, 12 Sep 2012 03:53:18 GMT

# POST to make metadata
$ curl -XPOST -Hx-container-meta-just-testing:some-value http://127.0.0.1:8080/v1/AUTH_test/test_container -Hx-auth-token:AUTH_tk71b0ad0c9b81435884091ae1ffea0ec1

# HEAD container to show the new metadata
$ curl -I http://127.0.0.1:8080/v1/AUTH_test/test_container -Hx-auth-token:AUTH_tk71b0ad0c9b81435884091ae1ffea0ec1
HTTP/1.1 204 No Content
X-Container-Object-Count: 0
X-Timestamp: 1347421947.14944
X-Container-Bytes-Used: 0
X-Container-Meta-Just-Testing: some-value
Accept-Ranges: bytes
Content-Length: 0
Date: Wed, 12 Sep 2012 03:54:43 GMT

# POST to remove the metadata item
$ curl -XPOST -Hx-remove-container-meta-just-testing:junk http://127.0.0.1:8080/v1/AUTH_test/test_container -Hx-auth-token:AUTH_tk71b0ad0c9b81435884091ae1ffea0ec1

# HEAD to show the metadata item is gone
$ curl -I http://127.0.0.1:8080/v1/AUTH_test/test_container -Hx-auth-token:AUTH_tk71b0ad0c9b81435884091ae1ffea0ec1
HTTP/1.1 204 No Content
X-Container-Object-Count: 0
X-Timestamp: 1347421947.14944
X-Container-Bytes-Used: 0
Accept-Ranges: bytes
Content-Length: 0
Date: Wed, 12 Sep 2012 03:55:29 GMT

Revision history for this message

Yukihiro KAWADA (warp-kawada) wrote on 2012-09-12:

OK.

I've got it.
Thank you very much for your sample.

-H 'X-Remove-Container-Meta-Web-Index:'
did not send header.
Because it is curl's feature.

Revision history for this message

Yukihiro KAWADA (warp-kawada) wrote on 2012-09-12:

#10

Now I understood what Alex said:

'X-Remove-Container-Meta-Web-Index: anything'

Revision history for this message

Yukihiro KAWADA (warp-kawada) wrote on 2012-09-12:

#11

Sorry, just one question.

curl -v -X POST -H 'X-Remove-Container-Read: junk'

does not work.

How do I remove X-Container-Read: ?

Thanks.

Revision history for this message

gholt (gholt) wrote on 2012-09-12:

#12

Ah, good catch. The -remove- form of deletion only works with meta headers and not with x-container-read and x-container-write. So in that case, this bug is valid. :)

That bug can be fixed in swift/proxy/controllers/base.py in the method Controller.transfer_headers

The idea is that you detect x-remove-container-read and x-remove-container-write and translate them to x-container-read and x-container-write headers with empty values, so they're sent that way to the backends. The backends don't need to know about the -remove- forms.

Changed in swift:
status:	Invalid → Confirmed

Revision history for this message

Alex Yang (alexyang) wrote on 2012-09-12:

#13

Yes. this is a bug. you are so careful.

Revision history for this message

Yukihiro KAWADA (warp-kawada) wrote on 2012-09-13:

#14

kwd_20120913.patch Edit (1.2 KiB, text/plain)

Thanks.

I've fixed.

Revision history for this message

Tom Fifield (fifieldt) wrote on 2012-10-07:

#15

This bug is still present in Folsom

(that is, not being able to reset ACLs)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-10-07: Fix proposed to swift (master)

#16

Fix proposed to branch: master
Review: https://review.openstack.org/14133

Changed in swift:
assignee:	nobody → Tom Fifield (fifieldt)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-10-11: Fix merged to swift (master)

#17

Reviewed: https://review.openstack.org/14133
Committed: http://github.com/openstack/swift/commit/124e75bc1bd0dd7eaeede5889d6a3441ad9694b1
Submitter: Jenkins
Branch: master

commit 124e75bc1bd0dd7eaeede5889d6a3441ad9694b1
Author: Tom Fifield <email address hidden>
Date: Sun Oct 7 14:28:41 2012 +1100

Allows removal of ACLs

fixes bug 1049017

As per the bug report, the -remove- form of deletion only works with
meta headers and not with x-container-read and x-container-write.

This patch by Yukihiro KAWADA detects the container acls and sends
them through to the backend empty like the other metadata values.

patch2 extends metadata-helper in ContainerController tests so that
the new functionality can be tested.

patch3 changes the k.lower().startswith() commands for read/write to
a single k.lower - thanks David :)

patch4 fixes PEP8

    patch5 fixes more than two hundred pep8 errors. Except one,
     where the pep8 tool complains of overindent, or underindent
    depending on the position on the line. No idea how to fix that one.

patch6 fixes the remaining pep8 error - thanks Darrell

Change-Id: I36c2dd72b4636136c5ce7db414bf2b61501090ad