[OSSA-2021-006] Routes middleware memory leak for nonexistent controllers (CVE-2021-40797)

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

This seems like it could represent a fairly quick means of memory consumption on servers where the API worker is running. I guess the impact is service degredation (as it starts to thrash swapspace) and eventually denial of service for the API once the kernel OOM killer gets involved and takes down the Python process?

@fungi - yes, it looks like that for me too. We are still investigating this issue with Rodolfo

After long investigation with Rodolfo on that issue, there is finally proposed fix/workaround for that memory leak in Neutron.

Hello:

Slawek found the place where Neutron was leaking memory. This is in the "routes.middleware.RoutesMiddleware" object. When using "singleton=True" (default value), a routes._RequestConfig() is always created [1]. This object has a thread safe variable to store the context information for each request.

The problem is that we use evenlet and we monkey patch the "threading" module (among many other ones) at the beginning of the Neutron server. The equivalent to "threading.local()" [1] is "eventlet.corolocal.local()". This object stores the greenthread-safe parameters in a WeakKeyDictionary() located in "eventlet.corolocal.local()._local__greens".

The problem we see is that this dictionary is not released. The patch provided in the next comment fixes this issue (this is just for testing, this patch could not land "routes" repository).

However, I think this is over engineering. The patch proposed by Slawek, using "singleton-=False" is a valid solution when using greenthreads.

Regards.

[1]https://github.com/bbangert/routes/blob/c4d5a5fb693ce8dc7cf5dbc591861acfc49d5c23/routes/middleware.py#L98
[2]https://github.com/bbangert/routes/blob/c4d5a5fb693ce8dc7cf5dbc591861acfc49d5c23/routes/__init__.py#L12

diff --git a/routes/middleware.py b/routes/middleware.py
index 649aa25..d06fd70 100644
--- a/routes/middleware.py
+++ b/routes/middleware.py
@@ -3,10 +3,12 @@ import re
 import logging

 from webob import Request
+from eventlet import greenthread

 from routes.base import request_config
 from routes.util import URLGenerator

+
 log = logging.getLogger('routes.middleware')

@@ -155,6 +157,9 @@ class RoutesMiddleware(object):
         # Wrapped in try as in rare cases the attribute will be gone already
         try:
             del self.mapper.environ
+ if self.singleton:
+ cur = greenthread.getcurrent()
+ del config._RequestConfig__shared_state._local__greens[cur]
         except AttributeError:
             pass
         return response

I'm not sure if we should make it public now, or should we treat it as private for now? I can propose fixes to stable branches here, that's not a problem at all but I think we should move on with that issue into some direction :)

This risk is only exploitable by authenticated users, right? With the worst impact from it being service degradation/denial of service, operators should be able to fairly quickly identify and revoke the malicious account. Given that, I think continuing to work this one in public would be a reasonable choice.

I agree, let's make and fix it publicly.

I checked with Rodoldo via IRC and he's okay with switching to our public security workflow for this, so I'll do that now.

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/807335

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807335
Committed: https://opendev.org/openstack/neutron/commit/e610a5eb9e71aa2549fb11e2139370d227787da2
Submitter: "Zuul (22348)"
Branch: master

commit e610a5eb9e71aa2549fb11e2139370d227787da2
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/807632

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/807633

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/807634

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/807635

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/807636

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/807637

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/807638

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807632
Committed: https://opendev.org/openstack/neutron/commit/c5e86f4f8f7e36233f383323e24e72dbe28efc04
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit c5e86f4f8f7e36233f383323e24e72dbe28efc04
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807633
Committed: https://opendev.org/openstack/neutron/commit/d961731a73128125ebf03fae52b4b34f3e7abf27
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit d961731a73128125ebf03fae52b4b34f3e7abf27
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807634
Committed: https://opendev.org/openstack/neutron/commit/2f642e23e085aa0bd5a23b4b169b436da9ad455c
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 2f642e23e085aa0bd5a23b4b169b436da9ad455c
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807637
Committed: https://opendev.org/openstack/neutron/commit/6d773bb8b84b496a71e4ee6c879af967aa031e98
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit 6d773bb8b84b496a71e4ee6c879af967aa031e98
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

As fixes have merged to all of Neutron's maintained stable branches, we can (and probably should) issue an advisory for this defect. I propose we use the following impact description to request a CVE assignment from MITRE, but please let me know if this isn't entirely accurate so I can correct it:

Title: Routes middleware memory leak for nonexistent controllers
Reporter: Slawek Kaplonski (Red Hat)
Products: Neutron
Affects: <16.4.1, >=17.0.0 <17.2.1, >=18.0.0 <18.1.1

Description:
Slawek Kaplonski with Red Hat reported a vulnerability in Neutron's routes middleware. By making API requests involving nonexistent controllers, an authenticated user may cause the API worker to consume increasing amounts of memory, resulting in API performance degradation or denial of service. All Neutron deployments are affected.

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807638
Committed: https://opendev.org/openstack/neutron/commit/dfbb9bcb5dae282c575b33171d691a5fca946beb
Submitter: "Zuul (22348)"
Branch: stable/queens

commit dfbb9bcb5dae282c575b33171d691a5fca946beb
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807635
Committed: https://opendev.org/openstack/neutron/commit/1977a8d9ba06e2361070f8894825ca388635c043
Submitter: "Zuul (22348)"
Branch: stable/train

commit 1977a8d9ba06e2361070f8894825ca388635c043
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807636
Committed: https://opendev.org/openstack/neutron/commit/b08715862e360c45cc0abd8341889e37858207bb
Submitter: "Zuul (22348)"
Branch: stable/stein

commit b08715862e360c45cc0abd8341889e37858207bb
Author: Slawek Kaplonski <email address hidden>
Date: Fri Sep 3 16:04:02 2021 +0200

    Don't use singleton in routes.middleware.RoutesMiddleware

    It seems that using default singleton=True in the
    routes.middleware.RoutesMiddleware which is leading to use thread-local
    RequestConfig singleton object is not working well with eventlet
    monkeypatching of threading library which we are doing in Neutron.
    As a result it leaks memory in neutron-api workers every time when API
    request to not existing API endpoint is made by user.

    To avoid that memory leak, let's use singletone=False in that
    RoutesMiddleware object, at least until problem with thread-local
    singleton and eventlet monkey patching will be solved.

    Closes-Bug: #1942179
    Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
    (cherry picked from commit e610a5eb9e71aa2549fb11e2139370d227787da2)

I checked with Slawek and Rodolfo in IRC, and they said they were good with the proposed impact description from comment #24, so I'm using that to submit a CVE request to MITRE now.

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ossa/+/807942

Reviewed: https://review.opendev.org/c/openstack/ossa/+/807942
Committed: https://opendev.org/openstack/ossa/commit/4f5d81b664a81ad7ba4856fbabe1d3f1f12a14e8
Submitter: "Zuul (22348)"
Branch: master

commit 4f5d81b664a81ad7ba4856fbabe1d3f1f12a14e8
Author: Jeremy Stanley <email address hidden>
Date: Wed Sep 8 20:15:03 2021 +0000

    Add OSSA-2021-006 (CVE-2021-40797)

    Change-Id: Ie61b5ffbec78e8c90e5ad773c9479f0d7ae1b932
    Closes-Bug: #1942179

Notifications have been distributed to the usual mailing lists.

This issue was fixed in the openstack/neutron 16.4.1 release.

This issue was fixed in the openstack/neutron 17.2.1 release.

This issue was fixed in the openstack/neutron 18.1.1 release.

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

sorry，i can't reproduce the bug in the neutron 17.2.0。Could you please give me any help about it?

Hello:

Slawek provided a method to test it. Actually we tested it, checking the Neutron server memory consumption increased linearly every time we made a HTTP request.

Deploy OpenStack Neutron (without the fix, now present in all versions) and execute this (changing your server IP address):
$ i=1; while [ $i -lt 2000 ]; do echo "Request $i"; curl -g -i -X GET http://10.120.0.30:9696/v2.0/blabla -H "Accept: application/json" -H "User-Agent: openstacksdk/0.59.0 keystoneauth1/4.3.1 python-requests/2.26.0 CPython/3.6.8" -H "X-Auth-Token: $token" 2>1 >/dev/null; i=$(( i+1 )); sleep 0.01; done

You'll see how the Neutron server memory grows continuously.

Regards

I deployed it by Kolla-ansible and tested it by the method.But the memory keep unchanged. The process as attached pictures below。

Maybe You already have fix for that issue included? Did You check that?

This issue was fixed in the openstack/neutron queens-eol release.

This issue was fixed in the openstack/neutron rocky-eol release.

This issue was fixed in the openstack/neutron stein-eol release.

This issue was fixed in the openstack/neutron train-eol release.