DragonFlow

df-controller.service start failed,due to OVS database connection to Open_vSwitch fail

Bug #1722839 reported by guanwenxue on 2017-10-11

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	DragonFlow	Fix Released	High	Omer Anson

Bug Description

Dragonflow already up-to-date,and deploy DF successful.

but <email address hidden> start failed.

service logs:

ERROR ovsdbapp.backend.ovs_idl [-] OVS database connection to Open_vSwitch failed with error: ''peer_bridge''. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: KeyError: 'peer_bridge'
ERROR ovsdbapp.backend.ovs_idl Traceback (most recent call last):
ERROR ovsdbapp.backend.ovs_idl File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 38, in start_connection
ERROR ovsdbapp.backend.ovs_idl cls.ovsdb_connection.start()
ERROR ovsdbapp.backend.ovs_idl File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 76, in start
ERROR ovsdbapp.backend.ovs_idl idlutils.wait_for_change(self.idl, self.timeout)
ERROR ovsdbapp.backend.ovs_idl File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 138, in wait_for_change
ERROR ovsdbapp.backend.ovs_idl while _idl.change_seqno == seqno and not _idl.run():
ERROR ovsdbapp.backend.ovs_idl File "/usr/local/lib/python2.7/dist-packages/ovs/db/idl.py", line 215, in run
ERROR ovsdbapp.backend.ovs_idl self.__parse_update(msg.result, OVSDB_UPDATE2)
ERROR ovsdbapp.backend.ovs_idl File "/usr/local/lib/python2.7/dist-packages/ovs/db/idl.py", line 434, in __parse_update
ERROR ovsdbapp.backend.ovs_idl self.__do_parse_update(update, version)
ERROR ovsdbapp.backend.ovs_idl File "/usr/local/lib/python2.7/dist-packages/ovs/db/idl.py", line 469, in __do_parse_update
ERROR ovsdbapp.backend.ovs_idl if self.__process_update2(table, uuid, row_update):
ERROR ovsdbapp.backend.ovs_idl File "/usr/local/lib/python2.7/dist-packages/ovs/db/idl.py", line 510, in __process_update2
ERROR ovsdbapp.backend.ovs_idl self.notify(ROW_CREATE, row)
ERROR ovsdbapp.backend.ovs_idl File "/opt/stack/dragonflow/dragonflow/ovsdb/impl_idl.py", line 94, in notify
ERROR ovsdbapp.backend.ovs_idl local_interface = ovs.OvsPort.from_idl_row(row)
ERROR ovsdbapp.backend.ovs_idl File "/opt/stack/dragonflow/dragonflow/db/models/ovs.py", line 82, in from_idl_row
ERROR ovsdbapp.backend.ovs_idl res.peer_bridge = row.external_ids['peer_bridge']
ERROR ovsdbapp.backend.ovs_idl KeyError: 'peer_bridge'
ERROR ovsdbapp.backend.ovs_idl
CRITICAL neutron [-] Unhandled error: OvsdbConnectionUnavailable: OVS database connection to Open_vSwitch failed with error: ''peer_bridge''. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
ERROR neutron Traceback (most recent call last):
ERROR neutron File "/usr/local/bin/df-local-controller", line 10, in <module>
ERROR neutron sys.exit(main())
ERROR neutron File "/opt/stack/dragonflow/dragonflow/cmd/eventlet/df_local_controller.py", line 17, in main
ERROR neutron df_local_controller.main()
ERROR neutron File "/opt/stack/dragonflow/dragonflow/controller/df_local_controller.py", line 352, in main
ERROR neutron controller.run()
ERROR neutron File "/opt/stack/dragonflow/dragonflow/controller/df_local_controller.py", line 89, in run
ERROR neutron self.vswitch_api.initialize(self.nb_api)
ERROR neutron File "/opt/stack/dragonflow/dragonflow/ovsdb/vswitch_impl.py", line 64, in initialize
ERROR neutron nb_api, db_connection, self.vsctl_timeout)
ERROR neutron File "/opt/stack/dragonflow/dragonflow/ovsdb/impl_idl.py", line 128, in __init__
ERROR neutron super(DFOvsdbApi, self).__init__(ovsdb_connection)
ERROR neutron File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 31, in __init__
ERROR neutron self.start_connection(connection)
ERROR neutron File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 43, in start_connection
ERROR neutron raise connection_exception
ERROR neutron OvsdbConnectionUnavailable: OVS database connection to Open_vSwitch failed with error: ''peer_bridge''. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
ERROR neutron

See original description

guanwenxue (guanwenxue) on 2017-10-11

summary:

- OvsdbConnectionUnavailable: OVS database connection to Open_vSwitch fail
+ df-controller.service start failed,due to OVS database connection to
+ Open_vSwitch fail

guanwenxue (guanwenxue) on 2017-10-15

description:

updated

Revision history for this message

Omer Anson (omer-anson) wrote on 2017-10-22:

Hi,

This is indeed a bug, and the information above should be enough for a fix.

Do you have more information? How did you reproduce? What local.conf file did you use?

Is it possible you had an additional patch port in the OVS bridges?

Thanks,
Omer.

Changed in dragonflow:
importance:	Undecided → High

Revision history for this message

guanwenxue (guanwenxue) wrote on 2017-10-23:

local.conf file Edit (1.4 KiB, text/plain)

Hi,
1.Just ./unstack.sh and ./stack.sh can reproduce
2.My local.conf file,please see the attachment
3.I think there are not another patch-port,but you can check:
root@Dragonflow:~# ovs-vsctl show
e5b6db9f-8e7e-416e-8c6c-671d3f0829ee
    Manager "ptcp:6640:0.0.0.0"
    Bridge br-ex
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth0"
            Interface "eth0"
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
        fail_mode: secure
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port gre-vtp
            Interface gre-vtp
                type: gre
                options: {key=flow, local_ip="192.168.1.113", remote_ip=flow}
        Port geneve-vtp
            Interface geneve-vtp
                type: geneve
                options: {key=flow, local_ip="192.168.1.113", remote_ip=flow}
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.6.1"

Omer Anson (omer-anson) on 2017-10-23

Changed in dragonflow:
assignee:	nobody → Omer Anson (omer-anson)

Revision history for this message

guanwenxue (guanwenxue) wrote on 2017-10-24:

Hi,
I reinstalled DF a moment ago,and found installation was failed, caused by the same error.

Revision history for this message

Irena Berezovsky (irenab) wrote on 2017-10-24:

Omer, even it is minor issue, but still weird. Check the error message above "CRITICAL neutron [-] Unhandled error: OvsdbConnectionUnavailable: OVS database connection to Open_vSwitch failed with error: ''peer_bridge''. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct."

It is ovn specific

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-12-27: Fix proposed to dragonflow (master)

Fix proposed to branch: master
Review: https://review.openstack.org/530196

Changed in dragonflow:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-12-27: Related fix proposed to dragonflow (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/530213

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-12-27: Fix merged to dragonflow (master)

Reviewed: https://review.openstack.org/530196
Committed: https://git.openstack.org/cgit/openstack/dragonflow/commit/?id=646525bf04397f4cebe27d6e586de1b2e6ef21c5
Submitter: Zuul
Branch: master

commit 646525bf04397f4cebe27d6e586de1b2e6ef21c5
Author: Omer Anson <email address hidden>
Date: Wed Dec 27 10:54:47 2017 +0200

Remove peer_bridge from OVS port

We have added peer_bridge to the OVS port, both in the model and the
OVSDB. This is Bad(TM) since:

1. It is not actually used

2. We have abused the Open_vSwitch OVSDB schema and added a field that
wasn't there.

3. The local controller fails and exits if this field doesn't exist on
all patch ports.

    This change removes that field. It is not strictly needed, and can be
    inferred from the information in the OVSDB. If this information is
    necessary, logic and helper code to do so will be added.

    Upgrade code is needed, but not urgent since this object is not actually
    kept in the NB API, only the OVSDB. This field will be ignored if it
    already exists.

Change-Id: Idc683694dedb50355bb109b285c052c38756e822
Closes-Bug: #1722839