object server deadlocks when a worker thread logs something
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| OpenStack Object Storage (swift) |
Fix Released
|
High
|
Samuel Merritt | ||
Bug Description
When code run in the object server's worker thread tries to log a message using Swift's logger, it can end up deadlocking the whole process.
Here's how the sad story unfolds:
The object server calls some code using tpool_reraise(), e.g. ECDiskFileWrite
When the message gets to the SysLogHandler, it goes to emit the record by sending it to a syslog server. The SysLogHandler has a mutex that it uses to serialize message emissions, so before it writes the record to a socket, it acquires the mutex.
The SysLogHandler's mutex is a threading.RLock object, but its internals have been monkeypatched by Eventlet. Instead of having a normal mutex that blocks the running pthread, it has a green mutex that only blocks the running greenthread. When some caller tries to acquire the green mutex but it is already held, the mutex object puts the current greenthread on a queue, then sleeps. When the lock holder releases the lock, it will wake up the head of the queue.
Finally, we get to the problem. Code in the worker pthread starts to log, but the SysLogHandler's mutex is held by another greenthread. The current mutex holder is a greenthread in the main pthread, i.e. the one doing WSGI stuff. So, the (one and only) greenthread in the worker pthread enqueues itself for wakeup later, then goes to sleep.
Now, the mutex-holder greenthread finishes its work, releases the lock, and goes to wake the worker pthread's greenthread... and KABOOM.
Traceback (most recent call last):
File "/usr/local/
timer()
File "/usr/local/
cb(*args, **kw)
File "/usr/local/
error: cannot switch to a different thread
A greenthread can only run in the pthread in which it was created. When the eventlet hub tries to run the worker pthread's greenthread, it fails. Worse, it leaves the mutex in a still-held state.
Now, for every request that comes in to this object server, a new greenthread is created (like normal), and when that greenthread goes to log, it blocks waiting for the busted mutex. This quickly exhausts available file descriptors as unclosed sockets pile up.
Meanwhile, the worker pthread's greenthread is still sleeping, waiting for someone to wake it up. If your tpool size happens to be 1, then this means any greenthread that tries to call tpool_execute will never receive a response.
One possible fix would be to use a lock that works across greenthreads in different pthreads.
| OpenStack Infra (hudson-openstack) wrote Fix proposed to swift (master) | #1 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to swift (master) | #2 |
| Changed in swift: | |
| status: | In Progress → Fix Released |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/swift 2.15.1 | #3 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to swift (stable/newton) | #4 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to swift (stable/ocata) | #5 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to swift (stable/newton) | #6 |
| tags: | added: in-stable-newton |
| tags: | added: in-stable-ocata |
| OpenStack Infra (hudson-openstack) wrote Fix merged to swift (stable/ocata) | #7 |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/swift ocata-eol | #8 |
Fix proposed to branch: master
Review: https:/