Comment 5 for bug 1691545

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/466420
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f4159d17552603b90912dba6fe5c604e8d0b8aa7
Submitter: Jenkins
Branch: stable/ocata

commit f4159d17552603b90912dba6fe5c604e8d0b8aa7
Author: melanie witt <email address hidden>
Date: Tue May 16 10:25:42 2017 +0000

    Cache database and message queue connection objects

    Recently in the gate we have seen a trace on some work-in-progress
    patches:

      OperationalError: (pymysql.err.OperationalError)
                        (1040, u'Too many connections')

    and at least one operator has reported that the number of database
    connections increased significantly going from Mitaka to Newton.

    It was suspected that the increase was caused by creating new oslo.db
    transaction context managers on-the-fly when switching database
    connections for cells. Comparing the dstat --tcp output of runs of the
    gate-tempest-dsvm-neutron-full-ubuntu-xenial job with and without
    caching of the database connections showed a difference of 445 active
    TCP connections and 1495 active TCP connections, respectively [1].

    This adds caching of the oslo.db transaction context managers and the
    oslo.messaging transports to avoid creating a large number of objects
    that are not being garbage-collected as expected.

    Closes-Bug: #1691545

    [1] https://docs.google.com/spreadsheets/d/1DIfFfX3kaA_SRoCM-aO7BN4IBEShChXLztOBFeKryt4/edit?usp=sharing

     Conflicts:
     nova/context.py
     nova/tests/unit/test_context.py

    NOTE(melwitt): Conflicts caused by the fact that the set_target_cell
    function doesn't exist in Ocata and message queue connections were
    not stored on the context in Ocata.

    Change-Id: I17e0eb836dd87aac5859f506e7d771d42753d31a
    (cherry picked from commit 47fa88d94754fcdad6bb132b45196b4d44c0f4cd)