mysqldb + eventlet = potential deadlocks

Related https://bugs.launchpad.net/neutron/+bug/1337690 ?

Yep, bug 1337690 is an example of this showing up in neutron.

This is a known problem plus we have an effort with switching to mysql-connector. Do you think we need this bug?

We deal with particular symptoms of it in correponding bugs.

I don't know - and I'm open to advice.
I'm creating an elastic-recheck query(*) and couldn't find an existing bug
that wasn't addressing a specific occurrence. I didn't know where to
create such an "openstack-wide" bug so opted for "oslo". Let me know if
there's a more suitable existing bug and/or project I should have used
instead.

(*) https://review.openstack.org/#/c/110531/

On 1 August 2014 21:04, Eugene Nikanorov <email address hidden> wrote:

> This is a known problem plus we have an effort with switching to mysql-
> connector. Do you think we need this bug?
>
> We deal with particular symptoms of it in correponding bugs.
>
> ** Changed in: neutron
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1350149
>
> Title:
> mysqldb + eventlet = potential deadlocks
>
> Status in OpenStack Neutron (virtual network service):
> Incomplete
> Status in Oslo - a Library of Common OpenStack Code:
> New
>
> Bug description:
> mysqldb is a C library, and thus can't be monkeypatched by eventlet.
>
> With eventlet, this series of events may lead to a deadlock:
> 1. greenlet A opens a mysql DB transaction, does some stuff
> 2. greenlet A yields, perhaps by making another unrelated blocking
> operation (eg: sending a network request)
> 3. greenlet B is scheduled and opens a mysql DB transaction that
> conflicts with A's
> 4. greenlet B enters mysqldb and the C call blocks, waiting until A
> completes
>
> Note that this is a deadlock on the client side, since A will never be
> scheduled again - and not a typical example of waiting for a
> transaction lock. The python process is now wedged until a mysqldb
> deadlock timer expires and B raises an exception.
>
> The mysql deadlock timer has a distinctive "Lock wait timeout
> exceeded; try restarting transaction" error message.
>
> There are bugs relating to specific instances of this, like bug 1298355
> and bug 1313794.
> .. but fundamentally this issue is generic across any use of mysqldb +
> eventlet + any code that yields the greenlet while holding a lock inside
> the database server. This bug is to track the generic issue across all
> such instances of the above pattern. Note it is not specific to oslo.db.
>
> See also this blueprint for a proposed fix (switch to a different mysql
> driver):
> - https://blueprints.launchpad.net/oslo/+spec/switch-to-mysql-connector
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/neutron/+bug/1350149/+subscriptions
>

--
 - Gus

ok, so the current vision of this situation is - 'MySQLdb' module doesn't patched by default - with ``eventlet.monkey_patch()`` (but 'psucopg2' does), so this type of mixed usage does not typically end well.

There was a mailing list thread discussing moving to PyMySQL: http://lists.openstack.org/pipermail/openstack-dev/2015-January/thread.html#55469

Fixed in oslo.db in https://review.openstack.org/#/c/184392/

Neutron fix (switch to pymysql) committed in https://review.openstack.org/#/c/184383/