MySQL error - too many connections

we see this in a lot of contexts downstream, and it turns out that based on load and such we do in fact need to raise the mysql_max_connections limit. Nova spawns a lot of Python processes, and each one can use anywhere from 1- 15 connections, 5 of which are persistently pooled.

I'm hearing on IRC that we're seeing it on Neutron as well and that the incidence of both seems to correspond. Basically, the source of the connections needs to be understood. I'd recommend using netstat to help figure it out, here's a line I used to identify individual processes:

    netstat -ntp | grep python | grep ESTABLISHED | grep 3306 | cut -c80-86 | sed 'sX/.*XX' | xargs -n1 ps -p | grep -v "CMD"

Most likely we need to tweak some settings my.cnf either in DevStack or DevStack-Gate.

Roman wrote a patch to gain more information: https://review.openstack.org/#/c/209858/

Yeah, I'm trying to figure out what's going on by periodically dumping information about connections to MySQL (https://review.openstack.org/#/c/209858/ - periodic task in nova-conductor).

I rechecked the patch multiple times - for most jobs the number of connections never exceeds ~150-200 units. So it's not clear to me how we actually managed to hit this problem (as in devstack the max connections number limit is set to 1024 - https://github.com/openstack-dev/devstack/blob/master/lib/databases/mysql#L99).

Curiously, according to logstash, there was another hit in the gate recently. But it was another Neutron job. I'm going to try something similar with Neutron too, as those jobs are only triggered on new changes to Neutron code.

what can happen is if something deadlocks hard in the DB, the other processes would pile up on top of it waiting. Though I've never really seen MySQL have a permanent deadlock before.

This is not really a nova job, it's unsurprising that we see this in neutron jobs, there are more services, so more connections. Though now that we're using the pure python mysql driver we might not need such a high API worker count

worker counts IMO (and in others O) need to come way down across openstack services. esp the worker-per-core model, on a 24 core machine we see all kinds of services spinning up dozens of DB connections (b.c. they are pooled) and then just sitting idle.