Looks like a core ovn issue. Still investigating but ovn-northd gets stuck retrying the same transaction (to set the FIP on the port) over and over and over:
2019-07-12T15:01:36.979Z|2796980|jsonrpc|DBG|unix:/usr/local/var/run/openvswitch/ovnsb_db.sock: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"},{"where":[["_uuid","==",["uuid","44c638c5-7b92-41e3-9a25-a1f4bf4c6ae1"]]],"row":{"nat_addresses":"fa:16:3e:de:ea:d5 172.24.5.169 is_chassis_resident(\"cr-lrp-01adeeb9-99f5-43ec-8da4-a7bb343c82f1\")"},"op":"update","table":"Port_Binding"},{"mutations":[["nat_addresses","insert",["set",["fa:16:3e:de:ea:d5 172.24.5.125 is_chassis_resident(\"cr-lrp-01adeeb9-99f5-43ec-8da4-a7bb343c82f1\")"]]]],"where":[["_uuid","==",["uuid","44c638c5-7b92-41e3-9a25-a1f4bf4c6ae1"]]],"op":"mutate","table":"Port_Binding"},{"where":[["_uuid","==",["uuid","104adb06-d1db-4617-95c6-3af720271e30"]]],"row":{"options":["map",[["ipv6_ra_address_mode","slaac"],["ipv6_ra_max_interval","600"],["ipv6_ra_min_interval","200"],["ipv6_ra_mtu","1392"],["ipv6_ra_prefixes","fd8b:814:608f::/64"],["ipv6_ra_send_periodic","true"],["ipv6_ra_src_addr","fe80::f816:3eff:fe8f:98c5"],["ipv6_ra_src_eth","fa:16:3e:8f:98:c5"],["peer","fa25893c-d654-47a2-bbee-f3acc89464c3"]]]},"op":"update","table":"Port_Binding"}], id=1456621
2019-07-12T15:01:36.979Z|2796981|poll_loop|DBG|wakeup due to [POLLIN] on fd 11 (<->/usr/local/var/run/openvswitch/ovnsb_db.sock) at lib/stream-fd.c:157 (83% CPU usage)
2019-07-12T15:01:36.979Z|2796982|jsonrpc|DBG|unix:/usr/local/var/run/openvswitch/ovnsb_db.sock: received reply, result=[{},{"count":1},{"count":1},{"count":1}], id=1456621
2019-07-12T15:01:36.979Z|2796983|poll_loop|DBG|wakeup due to 0-ms timeout at lib/ovsdb-idl.c:5397 (83% CPU usage)
2019-07-12T15:01:36.981Z|2796984|jsonrpc|DBG|unix:/usr/local/var/run/openvswitch/ovnsb_db.sock: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"},{"where":[["_uuid","==",["uuid","44c638c5-7b92-41e3-9a25-a1f4bf4c6ae1"]]],"row":{"nat_addresses":"fa:16:3e:de:ea:d5 172.24.5.169 is_chassis_resident(\"cr-lrp-01adeeb9-99f5-43ec-8da4-a7bb343c82f1\")"},"op":"update","table":"Port_Binding"},{"mutations":[["nat_addresses","insert",["set",["fa:16:3e:de:ea:d5 172.24.5.125 is_chassis_resident(\"cr-lrp-01adeeb9-99f5-43ec-8da4-a7bb343c82f1\")"]]]],"where":[["_uuid","==",["uuid","44c638c5-7b92-41e3-9a25-a1f4bf4c6ae1"]]],"op":"mutate","table":"Port_Binding"},{"where":[["_uuid","==",["uuid","104adb06-d1db-4617-95c6-3af720271e30"]]],"row":{"options":["map",[["ipv6_ra_address_mode","slaac"],["ipv6_ra_max_interval","600"],["ipv6_ra_min_interval","200"],["ipv6_ra_mtu","1392"],["ipv6_ra_prefixes","fd8b:814:608f::/64"],["ipv6_ra_send_periodic","true"],["ipv6_ra_src_addr","fe80::f816:3eff:fe8f:98c5"],["ipv6_ra_src_eth","fa:16:3e:8f:98:c5"],["peer","fa25893c-d654-47a2-bbee-f3acc89464c3"]]]},"op":"update","table":"Port_Binding"}], id=1456622
2019-07-12T15:01:36.981Z|2796985|poll_loop|DBG|wakeup due to [POLLIN] on fd 11 (<->/usr/local/var/run/openvswitch/ovnsb_db.sock) at lib/stream-fd.c:157 (83% CPU usage)
2019-07-12T15:01:36.981Z|2796986|jsonrpc|DBG|unix:/usr/local/var/run/openvswitch/ovnsb_db.sock: received reply, result=[{},{"count":1},{"count":1},{"count":1}], id=1456622
2019-07-12T15:01:36.981Z|2796987|poll_loop|DBG|wakeup due to 0-ms timeout at lib/ovsdb-idl.c:5397 (83% CPU usage)
2019-07-12T15:01:36.982Z|2796988|jsonrpc|DBG|unix:/usr/local/var/run/openvswitch/ovnsb_db.sock: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"},{"where":[["_uuid","==",["uuid","44c638c5-7b92-41e3-9a25-a1f4bf4c6ae1"]]],"row":{"nat_addresses":"fa:16:3e:de:ea:d5 172.24.5.169 is_chassis_resident(\"cr-lrp-01adeeb9-99f5-43ec-8da4-a7bb343c82f1\")"},"op":"update","table":"Port_Binding"},{"mutations":[["nat_addresses","insert",["set",["fa:16:3e:de:ea:d5 172.24.5.125 is_chassis_resident(\"cr-lrp-01adeeb9-99f5-43ec-8da4-a7bb343c82f1\")"]]]],"where":[["_uuid","==",["uuid","44c638c5-7b92-41e3-9a25-a1f4bf4c6ae1"]]],"op":"mutate","table":"Port_Binding"},{"where":[["_uuid","==",["uuid","104adb06-d1db-4617-95c6-3af720271e30"]]],"row":{"options":["map",[["ipv6_ra_address_mode","slaac"],["ipv6_ra_max_interval","600"],["ipv6_ra_min_interval","200"],["ipv6_ra_mtu","1392"],["ipv6_ra_prefixes","fd8b:814:608f::/64"],["ipv6_ra_send_periodic","true"],["ipv6_ra_src_addr","fe80::f816:3eff:fe8f:98c5"],["ipv6_ra_src_eth","fa:16:3e:8f:98:c5"],["peer","fa25893c-d654-47a2-bbee-f3acc89464c3"]]]},"op":"update","table":"Port_Binding"}], id=1456623
2019-07-12T15:01:36.983Z|2796989|poll_loop|DBG|wakeup due to [POLLIN] on fd 11 (<->/usr/local/var/run/openvswitch/ovnsb_db.sock) at lib/stream-fd.c:157 (83% CPU usage)
2019-07-12T15:01:36.983Z|2796990|jsonrpc|DBG|unix:/usr/local/var/run/openvswitch/ovnsb_db.sock: received reply, result=[{},{"count":1},{"count":1},{"count":1}], id=1456623
2019-07-12T15:01:36.983Z|2796991|poll_loop|DBG|wakeup due to 0-ms timeout at lib/ovsdb-idl.c:5397 (83% CPU usage)
Looks like a core ovn issue. Still investigating but ovn-northd gets stuck retrying the same transaction (to set the FIP on the port) over and over and over:
2019-07- 12T15:01: 36.979Z| 2796980| jsonrpc| DBG|unix: /usr/local/ var/run/ openvswitch/ ovnsb_db. sock: send request, method="transact", params= ["OVN_Southboun d",{"lock" :"ovn_northd" ,"op":" assert" },{"where" :[["_uuid" ,"==",[ "uuid", "44c638c5- 7b92-41e3- 9a25-a1f4bf4c6a e1"]]], "row":{ "nat_addresses" :"fa:16: 3e:de:ea: d5 172.24.5.169 is_chassis_ resident( \"cr-lrp- 01adeeb9- 99f5-43ec- 8da4-a7bb343c82 f1\")"} ,"op":" update" ,"table" :"Port_ Binding" },{"mutations" :[["nat_ addresses" ,"insert" ,["set" ,["fa:16: 3e:de:ea: d5 172.24.5.125 is_chassis_ resident( \"cr-lrp- 01adeeb9- 99f5-43ec- 8da4-a7bb343c82 f1\")"] ]]],"where" :[["_uuid" ,"==",[ "uuid", "44c638c5- 7b92-41e3- 9a25-a1f4bf4c6a e1"]]], "op":"mutate" ,"table" :"Port_ Binding" },{"where" :[["_uuid" ,"==",[ "uuid", "104adb06- d1db-4617- 95c6-3af720271e 30"]]], "row":{ "options" :["map" ,[["ipv6_ ra_address_ mode"," slaac"] ,["ipv6_ ra_max_ interval" ,"600"] ,["ipv6_ ra_min_ interval" ,"200"] ,["ipv6_ ra_mtu" ,"1392" ],["ipv6_ ra_prefixes" ,"fd8b: 814:608f: :/64"], ["ipv6_ ra_send_ periodic" ,"true" ],["ipv6_ ra_src_ addr"," fe80::f816: 3eff:fe8f: 98c5"], ["ipv6_ ra_src_ eth","fa: 16:3e:8f: 98:c5"] ,["peer" ,"fa25893c- d654-47a2- bbee-f3acc89464 c3"]]]} ,"op":" update" ,"table" :"Port_ Binding" }], id=1456621 12T15:01: 36.979Z| 2796981| poll_loop| DBG|wakeup due to [POLLIN] on fd 11 (<->/usr/ local/var/ run/openvswitch /ovnsb_ db.sock) at lib/stream-fd.c:157 (83% CPU usage) 12T15:01: 36.979Z| 2796982| jsonrpc| DBG|unix: /usr/local/ var/run/ openvswitch/ ovnsb_db. sock: received reply, result= [{},{"count" :1},{"count" :1},{"count" :1}], id=1456621 12T15:01: 36.979Z| 2796983| poll_loop| DBG|wakeup due to 0-ms timeout at lib/ovsdb- idl.c:5397 (83% CPU usage) 12T15:01: 36.981Z| 2796984| jsonrpc| DBG|unix: /usr/local/ var/run/ openvswitch/ ovnsb_db. sock: send request, method="transact", params= ["OVN_Southboun d",{"lock" :"ovn_northd" ,"op":" assert" },{"where" :[["_uuid" ,"==",[ "uuid", "44c638c5- 7b92-41e3- 9a25-a1f4bf4c6a e1"]]], "row":{ "nat_addresses" :"fa:16: 3e:de:ea: d5 172.24.5.169 is_chassis_ resident( \"cr-lrp- 01adeeb9- 99f5-43ec- 8da4-a7bb343c82 f1\")"} ,"op":" update" ,"table" :"Port_ Binding" },{"mutations" :[["nat_ addresses" ,"insert" ,["set" ,["fa:16: 3e:de:ea: d5 172.24.5.125 is_chassis_ resident( \"cr-lrp- 01adeeb9- 99f5-43ec- 8da4-a7bb343c82 f1\")"] ]]],"where" :[["_uuid" ,"==",[ "uuid", "44c638c5- 7b92-41e3- 9a25-a1f4bf4c6a e1"]]], "op":"mutate" ,"table" :"Port_ Binding" },{"where" :[["_uuid" ,"==",[ "uuid", "104adb06- d1db-4617- 95c6-3af720271e 30"]]], "row":{ "options" :["map" ,[["ipv6_ ra_address_ mode"," slaac"] ,["ipv6_ ra_max_ interval" ,"600"] ,["ipv6_ ra_min_ interval" ,"200"] ,["ipv6_ ra_mtu" ,"1392" ],["ipv6_ ra_prefixes" ,"fd8b: 814:608f: :/64"], ["ipv6_ ra_send_ periodic" ,"true" ],["ipv6_ ra_src_ addr"," fe80::f816: 3eff:fe8f: 98c5"], ["ipv6_ ra_src_ eth","fa: 16:3e:8f: 98:c5"] ,["peer" ,"fa25893c- d654-47a2- bbee-f3acc89464 c3"]]]} ,"op":" update" ,"table" :"Port_ Binding" }], id=1456622 12T15:01: 36.981Z| 2796985| poll_loop| DBG|wakeup due to [POLLIN] on fd 11 (<->/usr/ local/var/ run/openvswitch /ovnsb_ db.sock) at lib/stream-fd.c:157 (83% CPU usage) 12T15:01: 36.981Z| 2796986| jsonrpc| DBG|unix: /usr/local/ var/run/ openvswitch/ ovnsb_db. sock: received reply, result= [{},{"count" :1},{"count" :1},{"count" :1}], id=1456622 12T15:01: 36.981Z| 2796987| poll_loop| DBG|wakeup due to 0-ms timeout at lib/ovsdb- idl.c:5397 (83% CPU usage) 12T15:01: 36.982Z| 2796988| jsonrpc| DBG|unix: /usr/local/ var/run/ openvswitch/ ovnsb_db. sock: send request, method="transact", params= ["OVN_Southboun d",{"lock" :"ovn_northd" ,"op":" assert" },{"where" :[["_uuid" ,"==",[ "uuid", "44c638c5- 7b92-41e3- 9a25-a1f4bf4c6a e1"]]], "row":{ "nat_addresses" :"fa:16: 3e:de:ea: d5 172.24.5.169 is_chassis_ resident( \"cr-lrp- 01adeeb9- 99f5-43ec- 8da4-a7bb343c82 f1\")"} ,"op":" update" ,"table" :"Port_ Binding" },{"mutations" :[["nat_ addresses" ,"insert" ,["set" ,["fa:16: 3e:de:ea: d5 172.24.5.125 is_chassis_ resident( \"cr-lrp- 01adeeb9- 99f5-43ec- 8da4-a7bb343c82 f1\")"] ]]],"where" :[["_uuid" ,"==",[ "uuid", "44c638c5- 7b92-41e3- 9a25-a1f4bf4c6a e1"]]], "op":"mutate" ,"table" :"Port_ Binding" },{"where" :[["_uuid" ,"==",[ "uuid", "104adb06- d1db-4617- 95c6-3af720271e 30"]]], "row":{ "options" :["map" ,[["ipv6_ ra_address_ mode"," slaac"] ,["ipv6_ ra_max_ interval" ,"600"] ,["ipv6_ ra_min_ interval" ,"200"] ,["ipv6_ ra_mtu" ,"1392" ],["ipv6_ ra_prefixes" ,"fd8b: 814:608f: :/64"], ["ipv6_ ra_send_ periodic" ,"true" ],["ipv6_ ra_src_ addr"," fe80::f816: 3eff:fe8f: 98c5"], ["ipv6_ ra_src_ eth","fa: 16:3e:8f: 98:c5"] ,["peer" ,"fa25893c- d654-47a2- bbee-f3acc89464 c3"]]]} ,"op":" update" ,"table" :"Port_ Binding" }], id=1456623 12T15:01: 36.983Z| 2796989| poll_loop| DBG|wakeup due to [POLLIN] on fd 11 (<->/usr/ local/var/ run/openvswitch /ovnsb_ db.sock) at lib/stream-fd.c:157 (83% CPU usage) 12T15:01: 36.983Z| 2796990| jsonrpc| DBG|unix: /usr/local/ var/run/ openvswitch/ ovnsb_db. sock: received reply, result= [{},{"count" :1},{"count" :1},{"count" :1}], id=1456623 12T15:01: 36.983Z| 2796991| poll_loop| DBG|wakeup due to 0-ms timeout at lib/ovsdb- idl.c:5397 (83% CPU usage)
2019-07-
2019-07-
2019-07-
2019-07-
2019-07-
2019-07-
2019-07-
2019-07-
2019-07-
2019-07-
2019-07-
We're still investigating the issue.