Neighbor table entry for router missing with Linux bridge + L3HA + L2 population

After talking to Matt on IRC, the effect of this bug is that L3 HA + l2pop + LB is completely broken: This bug breaks connectivity between VMs and their router.

Matt did a lot of great isolation work, and found that with L3 HA + l2pop + OVS agent the L2 pop mech driver does send fdb info about the HA router ports:

http://paste.openstack.org/show/482246/

In that paste you can see that for HA router port 172.16.1.1, MAC fa:16:3e:1d:c6:5a, it's being sent and received. Also for the same MAC you can see two entries in table 20. Once learned via regular MAC learning (Via the flow in table 10), and once with l2pop. This is normal operation using the OVS agent.

When using LB, you can see in this paste:
http://paste.openstack.org/show/482242/

That the l2pop server side code is not sending an fdb entry about the HA router port.

Further investigation reveals inconsistency with reproducing this issue. When it occurs, the router port remains in the 'BUILD' state instead of changing to 'ACTIVE' which prevents the l2pop code from distributing the fdb entry to L2 agents.

amuller : I want to work on this. is two node devstack enough to test this out ?

At least three... two nodes with the L3 agent and one compute node.

ok I have two node devstack one has everything q-svc, q-l3, compute and second node q-l3, q-meta and ovs. logically i have two l3 servers. is that ok or i should add one more network node ?

This bug involves the Linux bridge agent, not OVS.

Is this bug fixed in Mitaka, or am I just lucky to not be able to reproduce it here?

Setup is as follows:

Router: .1, :1
DHCP agent 1: .2, :913b
DHCP agent 2: .3, :36a9
Instance: .4, :1296

On the network node running the active L3 agent and a DHCP agent, I see:

$ ip neigh show dev vxlan-65601
2a02:c0:4f0:1600:f816:3eff:fe5c:1296 lladdr fa:16:3e:5c:12:96 PERMANENT
2a02:c0:4f0:1600:f816:3eff:feb4:36a9 lladdr fa:16:3e:b4:36:a9 PERMANENT
192.168.0.4 lladdr fa:16:3e:5c:12:96 PERMANENT
192.168.0.3 lladdr fa:16:3e:b4:36:a9 PERMANENT

On the network node running a passive L3 agent and another DHCP agent:

$ ip neigh show dev vxlan-65601
2a02:c0:4f0:1600:f816:3eff:fe5c:1296 lladdr fa:16:3e:5c:12:96 PERMANENT
2a02:c0:4f0:1600:f816:3eff:feb4:36a9 lladdr fa:16:3e:b4:36:a9 PERMANENT
192.168.0.4 lladdr fa:16:3e:5c:12:96 PERMANENT
192.168.0.3 lladdr fa:16:3e:b4:36:a9 PERMANENT

On the compute node hosting the instance:

$ ip neigh show dev vxlan-65601
2a02:c0:4f0:1600:f816:3eff:feb4:36a9 lladdr fa:16:3e:b4:36:a9 PERMANENT
2a02:c0:4f0:1600:f816:3eff:fe31:913b lladdr fa:16:3e:31:91:3b PERMANENT
2a02:c0:4f0:1600::1 lladdr fa:16:3e:dd:b4:8c PERMANENT
192.168.0.1 lladdr fa:16:3e:c6:57:9f PERMANENT
192.168.0.2 lladdr fa:16:3e:31:91:3b PERMANENT
192.168.0.3 lladdr fa:16:3e:b4:36:a9 PERMANENT

Multicast doesn't work between the compute node and the two network nodes, and in any case "ip -d link show dev vxlan-65601" does not mention "group" at all.

It all seems to work just fine, but since it's not "supposed to" per this bug, I'm kind of wary of putting it into production until I fully understand why.

Running openstack-neutron-*-8.0.0-1.el7.noarch, for what it's worth.

I think I'm running into this problem in my hyper-converged environment with two network nodes. This is ultimately effecting our network architecture in our production environment(we also see this behavior there). We keep seeing multiple router interfaces getting stuck at the "BUILD" state. I'm not sure if it's because l2pop is not sending that fdb entry, but right now I'm not spinning up any instances..just making networks and attaching them to routers. Is there a way I could check to see if this is a problem with a lack of an FDB entry right now?. In the meantime...I think it's safe to say that my problem is related to this bug.
Below is a summary of the relevant environment configurations in my environment.

Two network nodes with these configurations in l3_agent.ini:
# HA failover
ha_confs_path = /var/lib/neutron/ha_confs
ha_vrrp_advert_int = 2
ha_vrrp_auth_password = 498d3d5737ca7273167c8badee533e9f2cf18e4db30676450
ha_vrrp_auth_type = PASS
handle_internal_only_routers = False
send_arp_for_ha = 3

in plugins/ml2/ml2_conf.ini:
mechanism_drivers = linuxbridge,l2population
l2_population = True

in plugins/ml2/linuxbridge_agent.ini:
[linux_bridge]

physical_interface_mappings = flat:eth12,vlan:eth11

# Linux bridge agent VXLAN networks
[vxlan]

enable_vxlan = True
vxlan_group = 239.1.1.1
# VXLAN local tunnel endpoint
local_ip = 172.29.243.52
l2_population = True

# Agent
[agent]
prevent_arp_spoofing = False

# Security groups
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver
enable_security_group = True

Like stated above, when we create our private tenant networks, then attach them to the router, some of them stay in the BUILD state until we bounce them(bring them down, then back up). Sometimes, the router gateway interfaces stay in the BUILD state too. Bouncing the interface doesn't always bring it back up the first time. Sometimes we have to bounce it multiple times before it finally reports as being "ACTIVE".
Keep in mind this is in an environment with 10 or so subnets being created rapidly. We are creating a pair of routers and many subnets using Ansible. All of this is done within the span of 30 seconds. Maybe we are creating them too fast?

Anyway, maybe someone here could help me figure out a workaround for now? Unfortunately, this isn't acceptable for our production environment.

Many thanks, and if there is anything I can do to help out with this bug please hit me up in IRC.

Turns out when I disable L2pop, my problem still persist. When I disable L3HA, my problem goes away. So my problem is with the l3ha vrrp driver.

I'm fairly certain the L3HA driver contains a bug (perhaps a race condition) that prevents changing the router state to active. The issue doesn't appear every time I create a router.

I've posted a new bug related to the issue I'm seeing since it is different from this bug:
https://bugs.launchpad.net/neutron/+bug/1590845

Status of this issue is unclear.

[Expired for neutron because there has been no activity for 60 days.]