Comment 5 for bug 1759971

* Localized it further to get_routers in l3 agent which does an RPC call to sync routers - setting a trace before that RPC call clearly makes the problem go away.

* And then to SQL query timing on the API side.

If I place a debug message printing the returned content,

https://github.com/openstack/neutron/blob/fbe308bdc12191c187343b5ef103dea9af738380/neutron/agent/l3/agent.py#L531-L534
            if update.action != queue.DELETE_ROUTER and not router:
                try:
                    update.timestamp = timeutils.utcnow()
                    routers = self.plugin_rpc.get_routers(self.context,
                                                          [update.id])
                    LOG.debug("Got routers via plugin_rpc: %s" % repr(routers)) # <--- added this to print the result without setting a trace above

https://github.com/openstack/neutron/blob/fbe308bdc12191c187343b5ef103dea9af738380/neutron/agent/l3/agent.py#L103-L107
        return cctxt.call(context, 'sync_routers', host=self.host,

I get the following output which does not contain address_scopes for the gateway port:

http://paste.openstack.org/raw/waUJmrtHFOqSnAr74mq3/

The code path on the API side:
neutron/api/rpc/handlers/l3_rpc.py: sync_routers

If I trace the whole sync_routers path from the beginning I get a proper result eventually:

http://paste.openstack.org/show/qGgkUzeFfeBpNM9hVhBL/

If I set a trace at the end of sync_routers (https://github.com/openstack/neutron/blob/fbe308bdc12191c187343b5ef103dea9af738380/neutron/api/rpc/handlers/l3_rpc.py#L129) then I get a reproducer:

https://paste.ubuntu.com/p/J5SJvc3xFF/

'gw_port': {'status': u'DOWN', 'created_at': '2018-04-04T14:07:02Z', 'binding:host_id': u'pillan', 'description': u'', 'allowed_address_pairs': [], 'tags': [], 'extra_dhcp_opts': [], 'updated_at': '2018-04-04T14:07:02Z', 'device_owner': u'network:router_gateway', 'revision_number': 3, 'port_security_enabled': False, 'binding:profile': {}, 'fixed_ips': [], 'id': u'7f0cff2e-a35c-463f-bcb1-1e2357db1bf0', 'security_groups': [], 'device_id': u'ef85572a-e3d0-49d6-a304-2468780b42cf', 'name': u'', 'admin_state_up': True, 'network_id': u'1b78176f-c608-4283-9bdc-e09961805e29', 'tenant_id': u'', 'binding:vif_details': {}, 'binding:vnic_type': u'normal', 'binding:vif_type': u'unbound', 'mac_address': u'fa:16:3e:7e:5e:42', 'project_id': u'', 'ip_allocation': u'deferred'},

If I compare it too a "good" gw_port dict, I can see that the status is "ACTIVE" so looks like when an RPC call to sync router dicts is made too fast and a gateway port is still down a different result is returned.

'gw_port': {'status': u'ACTIVE', 'created_at': '2018-04-04T13:15:37Z', 'binding:host_id': u'pillan', 'description': u'', 'allowed_address_pairs': [], 'tags': [], 'extra_dhcp_opts': [], 'updated_at': '2018-04-04T13:17:49Z', 'device_owner': u'network:router_gateway', 'revision_number': 7, 'port_security_enabled': False, 'binding:profile': {}, 'mtu': 1500, 'fixed_ips': [{'subnet_id': u'ee6795c0-8f06-4fa6-9531-04168403c74f', 'prefixlen': 21, 'ip_address': u'10.232.41.1'}], 'id': u'd6201e21-62e6-466c-b34c-3f342b1e5d3c', 'security_groups': [], 'device_id': u'beea4812-7d9d-4113-aa99-20db45a17d95', 'subnets': [{'dns_nameservers': [u'10.232.36.101'], 'ipv6_ra_mode': None, 'gateway_ip': '10.232.40.100', 'cidr': u'10.232.40.0/21', 'id': u'ee6795c0-8f06-4fa6-9531-04168403c74f', 'subnetpool_id': u'c7ba4af5-5aca-49e1-abbf-b7072d82d740'}], 'name': u'', 'admin_state_up': True, 'network_id': u'1b78176f-c608-4283-9bdc-e09961805e29', 'tenant_id': u'', 'binding:vif_details': {u'port_filter': True, u'datapath_type': u'system', u'ovs_hybrid_plug': True}, 'address_scopes': {4: u'd5d483bd-b1a1-4d11-8b98-a9697707321e', 6: None}, 'binding:vnic_type': u'normal', 'binding:vif_type': u'ovs', 'mac_address': u'fa:16:3e:a5:64:b5', 'project_id': u'', 'extra_subnets': [{'dns_nameservers': [u'10.232.36.101'], 'ipv6_ra_mode': None, 'gateway_ip': '10.232.16.1', 'cidr': u'10.232.16.0/21', 'id': u'59084723-8290-4cdd-996f-14de3f6eeacb', 'subnetpool_id': u'c7ba4af5-5aca-49e1-abbf-b7072d82d740'}], 'ip_allocation': u'deferred'},