OpenStack Object Storage (swift)

Unable to create a container

Bug #1051998 reported by Razique Mahroua
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Undecided
David Hadas
OpenStack Object Storage (swift) 1.9.0

Bug Description

Hi guys,
using swift SAIO (1.6.0-0ubuntu1~cloud0) with tempauth, swift configured and starting up with no error

/etc/swift/proxy.conf
http://paste.openstack.org/show/20948/

/etc/swift/account-server.conf
http://paste.openstack.org/show/20949/

/etc/swfit/container-server.conf
http://paste.openstack.org/show/20950/

/etc/swift/object-server.conf
http://paste.openstack.org/show/20951/

/etc/rsyncd.conf
http://paste.openstack.org/show/20952/

Commands :

$ curl -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://10.211.55.17:8080/auth/v1.0
* About to connect() to 10.211.55.17 port 8080 (#0)
* Trying 10.211.55.17... connected
> GET /auth/v1.0 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 10.211.55.17:8080
> Accept: */*
> X-Storage-User: test:tester
> X-Storage-Pass: testing
>
< HTTP/1.1 200 OK
< X-Storage-Url: http://127.0.0.1:8080/v1/AUTH_test
< X-Storage-Token: AUTH_tkc4eb1ff896de4dfbbe0740317d3d748b
< X-Auth-Token: AUTH_tkc4eb1ff896de4dfbbe0740317d3d748b
< Content-Length: 0
< Date: Fri, 14 Sep 2012 20:02:35 GMT
<
* Connection #0 to host 10.211.55.17 left intact
* Closing connection #0
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Now I try to create a container :

root@ubuntu-precise:~# curl -X PUT -H 'X-Auth-Token: AUTH_tkc4eb1ff896de4dfbbe0740317d3d748b' http://10.211.55.17:8080/v1/AUTH_test/mycontainer
202 Accepted

The request is accepted for processing.

Into /var/log/swift/storage(1&4)-error.log :

==> /var/log/swift/storage1.error <==
Sep 14 22:02:57 localhost container-server ERROR __call__ error with PUT /device/50918/AUTH_test/mycontainer : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/container/server.py", line 508, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1337, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.7/dist-packages/swift/container/server.py", line 233, in PUT#012 broker.initialize(timestamp)#012 File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 248, in initialize#012 'DB created by someone else while working?')#012DatabaseConnectionError: DB connection error (/srv/node1/device/containers/50918/1a4/31b98065d819a0bc43fa9e7f485bc1a4/31b98065d819a0bc43fa9e7f485bc1a4.db, 0):#012DB created by someone else while working? (txn: txc11a1d4f54fd42088826b924d1e73fa0)

==> /var/log/swift/storage4.error <==
Sep 14 22:02:57 localhost container-server ERROR __call__ error with PUT /device/50918/AUTH_test/mycontainer : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/container/server.py", line 508, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1337, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.7/dist-packages/swift/container/server.py", line 233, in PUT#012 broker.initialize(timestamp)#012 File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 248, in initialize#012 'DB created by someone else while working?')#012DatabaseConnectionError: DB connection error (/srv/node1/device/containers/50918/1a4/31b98065d819a0bc43fa9e7f485bc1a4/31b98065d819a0bc43fa9e7f485bc1a4.db, 0):#012DB created by someone else while working? (txn: txc11a1d4f54fd42088826b924d1e73fa0)

Time is synched, and not any ressources tries to access it. I recreated the rings and rebalanced them multiple times, no success

Best regards,
Razique

Revision history for this message
Constantine Peresypkin (constantine-q) wrote :

How many container servers do you have?
From the http://paste.openstack.org/show/20950/ it seems that you have only one.
It will never work this way.
Please see complete SAIO documentation on how it should be configured http://docs.openstack.org/developer/swift/development_saio.html

Revision history for this message
Razique Mahroua (razique) wrote :

Thanks Constantine, I've been able to resolve the issue, I was following the guide provided by CSSCorp, which stated to create three files
 /etc/swift/account-server.conf
 /etc/swift/object-server.conf
 /etc/swift/container-server.conf

amongst the /etc/swift/{container/object/account}-server/{1..4}.conf files

I removed the three files and updated my {1..4}.conf files

/etc/swift/account-server/1.conf
http://paste.openstack.org/show/20972/

(port 6022/ 6032/ 6042, updated devices and log facility for the three other files)

/etc/swift/container-server/1.conf
http://paste.openstack.org/show/20973/

/etc/swift/object-server/1.conf
http://paste.openstack.org/show/20974/

Now everything works fine

Revision history for this message
koolhead17 (koolhead17) wrote :

@Razique making it as Invalid as it was our configuration problem nothing upstream. :)

Changed in swift:
status: New → Invalid
Revision history for this message
David Kranz (david-kranz) wrote :
Download full text (4.0 KiB)

I just saw this in a cluster that has been running fine for a long time. Looking at the code, it is checking if a file exists and then later rechecking the existence, generating this error if the file now exists. But with multiple container-server processes running, this is a race condition that should be expected to happen sometimes. If this error matters, the code should really use create file inside a try rather than "check and create if not there". If the error is harmless it would help to make the message DEBUG or INFO rather than ERROR. Here is the log snippet for the transaction that had this error:

Jan 17 16:41:47 se06-1eth0 account-server 172.24.3.121 - - [17/Jan/2013:21:41:47 +0000] "HEAD /sdb1/20971/AUTH_94daee7efc9a4bd18170feac446d85d2" 204 - "txa02b236b1ce24efcbc4a901538336164" "-" "-" 0.0019 ""
Jan 17 16:41:47 se05-1 account-server 172.24.3.151 - - [17/Jan/2013:21:41:47 +0000] "PUT /sdb1/20971/AUTH_94daee7efc9a4bd18170feac446d85d2/unauthenticated_webtester_420fde67-54b0-41ea-9e3e-b976755783b6" 201 - "txa02b236b1ce24efcbc4a901538336164" "-" "-" 0.0016 ""
Jan 17 16:41:47 se07-1 account-server 172.24.3.161 - - [17/Jan/2013:21:41:47 +0000] "PUT /sdb1/20971/AUTH_94daee7efc9a4bd18170feac446d85d2/unauthenticated_webtester_420fde67-54b0-41ea-9e3e-b976755783b6" 201 - "txa02b236b1ce24efcbc4a901538336164" "-" "-" 0.0016 ""
Jan 17 16:41:47 se05-1 container-server 172.24.3.121 - - [17/Jan/2013:21:41:47 +0000] "PUT /sdc1/31409/AUTH_94daee7efc9a4bd18170feac446d85d2/unauthenticated_webtester_420fde67-54b0-41ea-9e3e-b976755783b6" 201 - "txa02b236b1ce24efcbc4a901538336164" "-" "-" 0.0100
Jan 17 16:41:47 se06-1eth0 container-server 172.24.3.121 - - [17/Jan/2013:21:41:47 +0000] "PUT /sdc1/31409/AUTH_94daee7efc9a4bd18170feac446d85d2/unauthenticated_webtester_420fde67-54b0-41ea-9e3e-b976755783b6" 201 - "txa02b236b1ce24efcbc4a901538336164" "-" "-" 0.0101
Jan 17 16:41:50 se08-1eth0 container-server ERROR __call__ error with PUT /sdb1/31409/AUTH_94daee7efc9a4bd18170feac446d85d2/unauthenticated_webtester_420fde67-54b0-41ea-9e3e-b976755783b6 : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/container/server.py", line 499, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1350, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.7/dist-packages/swift/container/server.py", line 232, in PUT#012 broker.initialize(timestamp)#012 File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 247, in initialize#012 'DB created by someone else while working?')#012DatabaseConnectionError: DB connection error (/srv/node/sdb1/containers/31409/0cf/f5630dac212afb5eb9a2912ffc9ae0cf/f5630dac212afb5eb9a2912ffc9ae0cf.db, 0):#012DB created by someone else while working? (txn: txa02b236b1ce24efcbc4a901538336164)
Jan 17 16:41:50 se08-1eth0 container-server 172.24.3.121 - - [17/Jan/2013:21:41:50 +0000] "PUT /sdb1/31409/AUTH_94daee7efc9a4bd18170feac446d85d2/unauthenticated_webtester_420fde67-54b0-41ea-9e3e-b976755783b6" 500 713 "txa02b236b1ce24efcbc4a901538336164" "-" "-" 2.7294
Jan 17 16:41:50 se02-1eth0 proxy-server Handoff requested (1) (txn: txa02...

Read more...

Changed in swift:
status: Invalid → New
Revision history for this message
David Hadas (david-hadas) wrote :

I think we can make it from an error to an handled condition
Once we find out that the db was created - we should continue as if it existed to begin with

Changed in swift:
assignee: nobody → David Hadas (david-hadas)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/28103

Changed in swift:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/28103
Committed: http://github.com/openstack/swift/commit/9745a28b18e085f1fc234db044402e320bd8085b
Submitter: Jenkins
Branch: master

commit 9745a28b18e085f1fc234db044402e320bd8085b
Author: David Hadas <email address hidden>
Date: Thu May 2 22:50:41 2013 +0300

    Handle DB creation race condition

    Till now if a race occurs on a request to a DB when it is
    first created, the loosing party die with 500...
    ... and an error stating a race is in the log file.

    Both removed by simply accepting the fact that the DB is in fact
    created by someone else - just less work for us to do... ;)

    Fixes Bug #1051998

    Change-Id: I8e29bf362edf2b1a84f9bffa39c04d5b3dc241b2

Changed in swift:
status: In Progress → Fix Committed
John Dickinson (notmyname)
Changed in swift:
milestone: none → 1.9.0
Thierry Carrez (ttx)
Changed in swift:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.