When using zmq, every event is processed two times

Bug #1585535 reported by duankebo
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
DragonFlow
Fix Released
Critical
Yuli

Bug Description

2016-05-24 12:53:33.827 22207 INFO dragonflow.db.api_nb [-] Pushing Update to Queue: Action:delete, Table:lport, Key:eff0afad-f3e5-4cc5-84d5-b2d86c20e0dd Value:None Topic:D
2016-05-24 12:53:33.828 22207 DEBUG dragonflow.db.api_nb [-] Event update: Action:delete, Table:lport, Key:eff0afad-f3e5-4cc5-84d5-b2d86c20e0dd Value:None Topic:D _read_db_changes_from_queue /opt/stack/new/dragonflow/dragonflow/db/api_nb.py:163
2016-05-24 12:53:33.828 22207 INFO dragonflow.db.api_nb [-] Pushing Update to Queue: Action:delete, Table:lport, Key:eff0afad-f3e5-4cc5-84d5-b2d86c20e0dd Value:None Topic:D
2016-05-24 12:53:33.829 22207 DEBUG dragonflow.db.api_nb [-] Event update: Action:delete, Table:lport, Key:eff0afad-f3e5-4cc5-84d5-b2d86c20e0dd Value:None Topic:D _read_db_changes_from_queue /opt/stack/new/dragonflow/dragonflow/db/api_nb.py:163
2016-05-24 12:53:34.090 22207 INFO dragonflow.db.api_nb [-] Pushing Update to Queue: Action:delete, Table:lswitch, Key:b1bf678e-5fd2-460b-87aa-51106560c32a Value:None Topic:D
2016-05-24 12:53:34.090 22207 DEBUG dragonflow.db.api_nb [-] Event update: Action:delete, Table:lswitch, Key:b1bf678e-5fd2-460b-87aa-51106560c32a Value:None Topic:D _read_db_changes_from_queue /opt/stack/new/dragonflow/dragonflow/db/api_nb.py:163
2016-05-24 12:53:34.091 22207 INFO dragonflow.controller.df_local_controller [-] Removing Logical Switch = {u'subnets': [], u'name': u'mynetwork1', u'router_external': False, u'topic': u'dd8c7aa1901a469b8fca9f31deb3a3e7', u'version': 4, u'id': u'b1bf678e-5fd2-460b-87aa-51106560c32a'}
2016-05-24 12:53:34.093 22207 INFO dragonflow.db.api_nb [-] Pushing Update to Queue: Action:delete, Table:lswitch, Key:b1bf678e-5fd2-460b-87aa-51106560c32a Value:None Topic:D
2016-05-24 12:53:34.093 22207 DEBUG dragonflow.db.api_nb [-] Event update: Action:delete, Table:lswitch, Key:b1bf678e-5fd2-460b-87aa-51106560c32a Value:None Topic:D _read_db_changes_from_queue /opt/stack/new/dragonflow/dragonflow/db/api_nb.py:163
2016-05-24 12:53:34.093 22207 INFO dragonflow.controller.df_local_controller [-] Removing Logical Switch = None
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb [-] 'NoneType' object has no attribute 'get_id'
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb Traceback (most recent call last):
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb File "/opt/stack/new/dragonflow/dragonflow/db/api_nb.py", line 175, in _read_db_changes_from_queue
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb value)
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb File "/opt/stack/new/dragonflow/dragonflow/db/api_nb.py", line 231, in apply_db_change
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb self.controller.logical_switch_deleted(lswitch_id)
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb File "/opt/stack/new/dragonflow/dragonflow/controller/df_local_controller.py", line 187, in logical_switch_deleted
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb self.open_flow_app.notify_remove_logical_switch(lswitch)
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb File "/opt/stack/new/dragonflow/dragonflow/controller/ryu_base_app.py", line 84, in notify_remove_logical_switch
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb self.dispatcher.dispatch('remove_logical_switch', lswitch=lswitch)
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb File "/opt/stack/new/dragonflow/dragonflow/controller/dispatcher.py", line 43, in dispatch
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb handler(*args, **kwargs)
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb File "/opt/stack/new/dragonflow/dragonflow/controller/l2_app.py", line 333, in remove_logical_switch
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb lswitch.get_id(),
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb AttributeError: 'NoneType' object has no attribute 'get_id'
2016-05-24 12:53:34.093 22207 ERROR dragonflow.db.api_nb

Revision history for this message
Li Ma (nick-ma-z) wrote :

I also noticed this problem and it causes many random failure in the ci.

Changed in dragonflow:
importance: Undecided → Critical
Yuli (stremovsky)
Changed in dragonflow:
assignee: nobody → Yuli (stremovsky)
Revision history for this message
duankebo (duankebo) wrote :

Hi Yuli, do you know the root cause of this problem?

Revision history for this message
hujie (mike-hu) wrote :

Did you see the plugin side? Is it only publish one msg?

Revision history for this message
Yuli (stremovsky) wrote :

Hello

In http://logs.openstack.org/15/327515/4/check/gate-dragonflow-dsvm-fullstack-nv/ff2b8dd/logs/screen-q-svc.txt.gz
I see 2 the following message: about to connect to IPC socket

It might happened because 2 connections were opened with zmq.

I am continue researching this topic.

Yuli

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to dragonflow (master)

Fix proposed to branch: master
Review: https://review.openstack.org/332673

Changed in dragonflow:
status: New → In Progress
Revision history for this message
duankebo (duankebo) wrote :

I have checked the log of dragonflow, the problem has been solved. Congratulations!

Revision history for this message
Yuli (stremovsky) wrote :

Good news!

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to dragonflow (master)

Reviewed: https://review.openstack.org/332673
Committed: https://git.openstack.org/cgit/openstack/dragonflow/commit/?id=8adce333b7d2391cde6eebf4c940f0e088923401
Submitter: Jenkins
Branch: master

commit 8adce333b7d2391cde6eebf4c940f0e088923401
Author: yuli <email address hidden>
Date: Wed Jun 22 10:43:05 2016 +0300

    Fix receving double messages in df-controller

    No need to configure publishers_ips variable by default.
    We fetch publisher ips from the database.

    Closes-bug #1585535

    Change-Id: I2439ecb76bb9100bfff430ecae659f7e162c18ab

Changed in dragonflow:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.