Zun

too many connections from zun-api to rabbitmq

Bug #1785371 reported by Wesley Gao
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Zun
Fix Released
Critical
hongbin
Queens
Fix Committed
Critical
hongbin

Bug Description

I was doing performance test on Queens. When I counted the number of connections to rabbitmq, found that zun-api's count was too large.

1 nova-consoleaut
3 nova-scheduler
4 nova-compute
4 zun-compute
5 neutron-dhcp-ag
7 neutron-l3-agen
11 neutron-linuxbr
42 neutron-metadat
80 nova-api
171 nova-conductor
329 neutron-server
3580 zun-api

And it increases as I invoke the zunclient, but never gets down.

Here is my fix:

diff --git a/zun/zun/api/hooks.py b/zun/zun/api/hooks.py
index 1f2d43b..ef25881 100644
--- a/zun/zun/api/hooks.py
+++ b/zun/zun/api/hooks.py
@@ -80,6 +80,8 @@ class RPCHook(hooks.PecanHook):
         context = state.request.context
         state.request.compute_api = compute_api.API(context)

+ def after(self, state):
+ state.request.compute_api.rpcapi.cleanup()

 class NoExceptionTracebackHook(hooks.PecanHook):
     """Workaround rpc.common: deserialize_remote_exception.
diff --git a/zun/zun/common/rpc_service.py b/zun/zun/common/rpc_service.py
index 157ffb4..2f3656b 100644
--- a/zun/zun/common/rpc_service.py
+++ b/zun/zun/common/rpc_service.py
@@ -112,3 +112,6 @@ class API(object):

     def echo(self, message):
         self._cast('echo', message=message)
+
+ def cleanup(self):
+ self._client.transport.cleanup()

qyq (qianyuqiao)
Changed in zun:
assignee: nobody → 潜禹桥 (qianyuqiao)
assignee: 潜禹桥 (qianyuqiao) → nobody
hongbin (hongbin034)
Changed in zun:
status: New → Triaged
importance: Undecided → High
hongbin (hongbin034)
Changed in zun:
importance: High → Critical
assignee: nobody → hongbin (hongbin034)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to zun (master)

Fix proposed to branch: master
Review: https://review.openstack.org/588992

Changed in zun:
status: Triaged → In Progress
Revision history for this message
hongbin (hongbin034) wrote :

Thanks for reporting the bug. I analyzed how other projects (i.e. nova) manage the RPC connections and borrowed their idea of using a global RPC transport per API server. This should further improve the performance since we don't have to create/cleanup the RPC connection per API call.

Here is the proposed fix: https://review.openstack.org/#/c/588992/ . Let me know if there is any feedback.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to zun (master)

Reviewed: https://review.openstack.org/588992
Committed: https://git.openstack.org/cgit/openstack/zun/commit/?id=de82a5dc81816194eecbbf609ca3c179b7aa9368
Submitter: Zuul
Branch: master

commit de82a5dc81816194eecbbf609ca3c179b7aa9368
Author: Hongbin Lu <email address hidden>
Date: Sun Aug 5 20:28:57 2018 +0000

    Reuse the RPC transport to create RPC client

    In before, we created a PRC transport per RPC call. This causes
    too many RPC connections as reported by the bug. This patch fixes
    it by initialize the RPC transport at startup and reuse the RPC
    transport for each API call.

    Change-Id: I55e409e0b70024673bd25ab460c2a5bfecc717a5
    Closes-Bug: #1785371

Changed in zun:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to zun (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/589331

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to zun (stable/queens)

Reviewed: https://review.openstack.org/589331
Committed: https://git.openstack.org/cgit/openstack/zun/commit/?id=db0cea7121628d6d27e0beaa10b989be33dad37e
Submitter: Zuul
Branch: stable/queens

commit db0cea7121628d6d27e0beaa10b989be33dad37e
Author: Hongbin Lu <email address hidden>
Date: Sun Aug 5 20:28:57 2018 +0000

    Reuse the RPC transport to create RPC client

    In before, we created a PRC transport per RPC call. This causes
    too many RPC connections as reported by the bug. This patch fixes
    it by initialize the RPC transport at startup and reuse the RPC
    transport for each API call.

    Change-Id: I55e409e0b70024673bd25ab460c2a5bfecc717a5
    Closes-Bug: #1785371
    (cherry picked from commit de82a5dc81816194eecbbf609ca3c179b7aa9368)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/zun 2.1.0

This issue was fixed in the openstack/zun 2.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/zun 1.0.1

This issue was fixed in the openstack/zun 1.0.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.