redis driver raise exception when node removed

Bug #1642527 reported by wang jian
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
DragonFlow
Fix Released
High
feipeng

Bug Description

set enable_port_status_notifier=True for neutron-server, then remove a redis node
2016-11-17 17:37:19.847 115759 WARNING dragonflow.db.pubsub_drivers.redis_db_pubsub_driver [req-bdb4503b-73e5-4362-98ae-b56b66de9dd3 - -] subscriber listening task lost connection Error 111 connecting to 10.248.92.227:4002. ECONNREFUSED.
2016-11-17 17:37:19.848 115759 INFO dragonflow.db.drivers.redis_mgt [req-bdb4503b-73e5-4362-98ae-b56b66de9dd3 - -] remove node 10.248.92.227:4002 from redis master list
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
    func(*args, **kwargs)
  File "/opt/stack/dragonflow/dragonflow/db/pubsub_drivers/redis_db_pubsub_driver.py", line 239, in run
    self.register_hamsg_for_db()
  File "/opt/stack/dragonflow/dragonflow/db/pubsub_drivers/redis_db_pubsub_driver.py", line 179, in register_hamsg_for_db
    self.redis_mgt.register_ha_topic()
  File "/opt/stack/dragonflow/dragonflow/db/drivers/redis_mgt.py", line 334, in register_ha_topic
    if self.subscriber is not None:
AttributeError: 'RedisMgt' object has no attribute 'subscriber'
2016-11-17 17:37:22.636 115759 INFO dragonflow.db.pubsub_drivers.redis_db_pubsub_driver [-] publish connection old masterlist [{'ip_port': '10.248.92.227:4001', 'slot': [[0, 5460]]}, {'ip_port': '10.248.92.227:4003', 'slot': [[10923, 16383]]}]
2016-11-17 17:37:22.637 115761 WARNING dragonflow.db.drivers.redis_mgt [-] redis cluster nodes less than local, maybe there is a partition in db cluster, nodes:{'10.248.92.227:4001': {'status': 'connected', 'last_pong_rcvd': '0', 'epoch': '1', 'node_id': '023d96311c63155a039cc405a3a6d039d17ac7fc', 'role': 'master', 'last_ping_sent': '0', 'slots': [['0', '5460']], 'master_id': '-'}, '10.248.92.227:4003': {'status': 'connected', 'last_pong_rcvd': '1479375437618', 'epoch': '3', 'node_id': 'c05794146cf2beffe6ca3f51ed673829e4711ec1', 'role': 'master', 'last_ping_sent': '0', 'slots': [['10923', '16383']], 'master_id': '-'}, '10.248.92.227:4002': {'status': 'disconnected', 'last_pong_rcvd': '1479375439649', 'epoch': '2', 'node_id': '1db300860fe8bffa9019e1a5185fc29821e6c44f', 'role': 'master', 'last_ping_sent': '1479375439926', 'slots': [['5461', '10922']], 'master_id': '-'}, '10.248.92.227:4004': {'status': 'connected', 'last_pong_rcvd': '1479375441633', 'epoch': '4', 'node_id': '4f982c8410505830ea0fe5b99bddeeadd111a17d', 'role': 'slave', 'last_ping_sent': '0', 'slots': [], 'master_id': '023d96311c63155a039cc405a3a6d039d17ac7fc'}, '10.248.92.227:4006': {'status': 'connected', 'last_pong_rcvd': '1479375441131', 'epoch': '6', 'node_id': '704857fbe4665e099d5fc7e8d3b48e8f6e64309c', 'role': 'slave', 'last_ping_sent': '0', 'slots': [], 'master_id': 'c05794146cf2beffe6ca3f51ed673829e4711ec1'}}, local nodes:{'10.248.92.227:4001': {'status': 'connected', 'last_pong_rcvd': '0', 'epoch': '1', 'node_id': '023d96311c63155a039cc405a3a6d039d17ac7fc', 'role': 'master', 'last_ping_sent': '0', 'slots': [['0', '5460']], 'master_id': '-'}, '10.248.92.227:4003': {'status': 'connected', 'last_pong_rcvd': '1479375234189', 'epoch': '3', 'node_id': 'c05794146cf2beffe6ca3f51ed673829e4711ec1', 'role': 'master', 'last_ping_sent': '0', 'slots': [['10923', '16383']], 'master_id': '-'}, '10.248.92.227:4002': {'status': 'connected', 'last_pong_rcvd': '1479375232183', 'epoch': '2', 'node_id': '1db300860fe8bffa9019e1a5185fc29821e6c44f', 'role': 'master', 'last_ping_sent': '0', 'slots': [['5461', '10922']], 'master_id': '-'}, '10.248.92.227:4005': {'status': 'connected', 'last_pong_rcvd': '1479375235191', 'epoch': '5', 'node_id': '2e4d50cc44daf56324fa21595cdfe18c2bd1e4c5', 'role': 'slave', 'last_ping_sent': '0', 'slots': [], 'master_id': '1db300860fe8bffa9019e1a5185fc29821e6c44f'}, '10.248.92.227:4004': {'status': 'connected', 'last_pong_rcvd': '1479375231180', 'epoch': '4', 'node_id': '4f982c8410505830ea0fe5b99bddeeadd111a17d', 'role': 'slave', 'last_ping_sent': '0', 'slots': [], 'master_id': '023d96311c63155a039cc405a3a6d039d17ac7fc'}, '10.248.92.227:4006': {'status': 'connected', 'last_pong_rcvd': '1479375233186', 'epoch': '6', 'node_id': '704857fbe4665e099d5fc7e8d3b48e8f6e64309c', 'role': 'slave', 'last_ping_sent': '0', 'slots': [], 'master_id': 'c05794146cf2beffe6ca3f51ed673829e4711ec1'}}
2016-11-17 17:37:22.637 115759 INFO dragonflow.db.pubsub_drivers.redis_db_pubsub_driver [-] publish connection new masterlist [{'ip_port': '10.248.92.227:4001', 'slot': [[0, 5460]]}, {'ip_port': '10.248.92.227:4003', 'slot': [[10923, 16383]]}]

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to dragonflow (master)

Fix proposed to branch: master
Review: https://review.openstack.org/399334

Li Ma (nick-ma-z)
Changed in dragonflow:
importance: Undecided → Medium
Omer Anson (omer-anson)
Changed in dragonflow:
importance: Medium → High
assignee: nobody → feipeng (feipeng1)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to dragonflow (master)

Reviewed: https://review.openstack.org/399334
Committed: https://git.openstack.org/cgit/openstack/dragonflow/commit/?id=ab9680251bf120ab8322614b3c1bf7b511c797cd
Submitter: Jenkins
Branch: master

commit ab9680251bf120ab8322614b3c1bf7b511c797cd
Author: feipeng <email address hidden>
Date: Fri Nov 18 02:18:56 2016 +0000

    fix problem about subscriber on neutron server when redis failover

    subscriber can be initialized on neutron,when redis failover,it
    may cause err as bug described

    Change-Id: If3582a847f452b09c6cdcce30136f86f327dd803
    Closes-bug: #1642527

Changed in dragonflow:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.