FWaaS: Firewall creation fails in case of distributed routers (Pike)

Bug #1715395 reported by Christoph Fiehe
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Swaminathan Vasudevan

Bug Description

I have manually setup a fresh OpenStack Pike HA environment based on Ubuntu 16.04.3 in conjunction with DVR. Firewall creation works in case of centralized routers, but when a firewall gets attached to a distributed router, the firewall gets stuck in "PENDUNG UPDATE". The log file contains the following exception:

2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server [req-28e7a23e-fa55-4358-9977-c1db08435624 dddfba8e02f746799a6408a523e6cd25 ed2d2efd86dd40e7a45491d8502318d3 - - -] Exception during message handling: AttributeError: 'DvrEdgeHaRouter' object has no attribute 'dist_fip_count'
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_log/helpers.py", line 67, in wrapper
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent.py", line 284, in create_firewall
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server firewall)
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas.py", line 89, in create_firewall
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server self._setup_firewall(agent_mode, apply_list, firewall)
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas.py", line 195, in _setup_firewall
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server agent_mode, router_info)
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas.py", line 119, in _get_ipt_mgrs_with_if_prefix
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server if router_info.dist_fip_count:
2017-09-06 13:58:29.572 22581 ERROR oslo_messaging.rpc.server AttributeError: 'DvrEdgeHaRouter' object has no attribute 'dist_fip_count'

Some version information:
$ pip list | grep neutron
neutron (11.0.0)
neutron-fwaas (11.0.0)
neutron-fwaas-dashboard (1.0.1.dev1)
neutron-lbaas (11.0.0)
neutron-lbaas-dashboard (3.0.1)
neutron-lib (1.9.1)

##############################
l3_agent.ini
##############################

[DEFAULT]
agent_mode = dvr_snat
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

[agent]
extensions = fwaas

[fwaas]
agent_version = v1
driver = iptables
enabled = true

##############################
neutron.conf
##############################

[DEFAULT]
allow_overlapping_ips = true
auth_strategy = keystone
base_mac = 02:05:69:00:00:00
bind_host = 10.30.200.101
bind_port = 9696
core_plugin = ml2
debug = false
default_log_levels=amqp=WARN,amqplib=WARN,boto=WARN,qpid=WARN,sqlalchemy=WARN,suds=WARN,oslo.messaging=WARN,iso8601=WARN,requests.packages.urllib3.connectionpool=WARN,urllib3.connectionpool=WARN,websocket=WARN,requests.packages.urllib3.util.retry=WARN,urllib3.util.retry=WARN,keystonemiddleware=WARN,routes.middleware=WARN,stevedore=WARN,taskflow=WARN,keystoneauth=WARN,oslo.cache=WARN,dogpile.core.dogpile=WARN,oslo_service=WARN,neutron=WARN
dhcp_agents_per_network = 2
dns_domain = openstack.mycompany.com.
dvr_base_mac = 0A:05:69:00:00:00
endpoint_type = internalURL
host = os-network01
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
l3_ha = true
l3_ha_net_cidr = 169.254.192.0/18
log_dir = /var/log/neutron
max_l3_agents_per_router = 2
min_l3_agents_per_router = 2
notify_nova_on_port_data_changes = true
notify_nova_on_port_status_changes = true
router_distributed = true
service_plugins = router,firewall,qos,lbaasv2
state_path = /var/lib/neutron
transport_url = rabbit://neutron:neutronpass@os-rabbit01:5672,neutron:neutronpass@os-rabbit02:5672/openstack

[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf

[database]
connection = mysql+pymysql://neutron:neutronDBpass@os-controller/neutron
max_retries = -1

[keystone_authtoken]
auth_type = password
auth_uri = https://os-cloud.mycompany.com:5000
auth_url = http://os-identity:35357
memcached_servers = os-memcache:11211
password = neutronpass
project_domain_name = default
project_name = service
user_domain_name = default
username = neutron

[nova]
auth_type = password
auth_url = http://os-identity:35357
endpoint_type = internal
password = novapass
project_domain_name = default
project_name = service
region_name = RegionOne
user_domain_name = default
username = nova

[oslo_concurrency]
lock_path = /var/lock/neutron

[oslo_messaging_notifications]
driver = messagingv2

[oslo_messaging_rabbit]
amqp_durable_queues = true
rabbit_ha_queues = true
rabbit_retry_backoff = 2
rabbit_retry_interval = 1

[oslo_middleware]
enable_proxy_headers_parsing = true

[service_providers]
service_provider = FIREWALL:Iptables:neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver:default
service_provider = LOADBALANCERV2:Haproxy:neutron_lbaas.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default

##############################
fwaas_driver.ini
##############################

[fwaas]
enabled = true
driver = neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas.IptablesFwaasDriver

May someone please have a look.

Revision history for this message
Brian Haley (brian-haley) wrote :

While the neutron base code changed in April 2017, the FWaaS code did not, so there's a bug.

The code needs to change something like this:

< if router_info.dist_fip_count:
--
> if router_info.rtr_fip_connect:

Changed in neutron:
status: New → Confirmed
importance: Undecided → High
Changed in neutron:
assignee: nobody → Reedip (reedip-banerjee)
Revision history for this message
Reedip (reedip-banerjee-deactivatedaccount) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-fwaas (master)

Fix proposed to branch: master
Review: https://review.openstack.org/501570

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
Christoph Fiehe (fiehe) wrote :

Thank you very much for the patch. I made a quick hack and can confirm that your fix is working. Firewalls now get attached and work properly in case of distributed routers.

Unfortunaetly, when you remove a firewall from a router or update some of its rules when the firewall is ACTIVE, e.g. change "DENY" -> "ALLOW", the modification has no effect. It seems that the iptables rules are not removed and updated, respectively. The log does not contain any error entries.

Can you please check this?

Revision history for this message
Reedip (reedip-banerjee-deactivatedaccount) wrote :

@Jens: I think what you mean is that if any Firewall which has a set of rules, is updated, then iptables is not affected. Its not related to DVR?

Revision history for this message
Christoph Fiehe (fiehe) wrote :

Exactly, updating a Firewall attached to a centralized Router works without any issues, but update fails in case of a distributed router.

Revision history for this message
Reedip (reedip-banerjee-deactivatedaccount) wrote :

@Jens , I think we need a separate bug for tracking this issue

Revision history for this message
Reedip (reedip-banerjee-deactivatedaccount) wrote :

We also need some logs , or some snippets :)

tags: added: l3-dvr-backlog
Revision history for this message
Reedip (reedip-banerjee-deactivatedaccount) wrote :
Revision history for this message
Christoph Fiehe (fiehe) wrote :

@Reedip, sorry I have not noticed that you have already raised an issue. I have just raised Bug #1716401. I have added some information showing what is happening (or better not happening) on a compute node in case of FWaaS with DVR. We can mark my issue as duplicate.

Changed in neutron:
assignee: Reedip (reedip-banerjee) → Swaminathan Vasudevan (swaminathan-vasudevan)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-fwaas (master)

Reviewed: https://review.openstack.org/501570
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=5706252c7947887e3c5b00f258dd847027497b97
Submitter: Zuul
Branch: master

commit 5706252c7947887e3c5b00f258dd847027497b97
Author: Reedip <email address hidden>
Date: Thu Sep 7 05:36:11 2017 +0000

    DVR-FWaaS: Fix DVR FWaaS rules for fipnamespace

    FWaaS for DVR routers were only supported for
    North-South traffic. But with the recent change
    in the DVR router-info to handle FloatingIP's
    as mentioned in the reference patch[1], the
    dist_fip_count variable was removed and was
    replaced by rtr_fip_connect.

    So change in variable 'rtr_fip_connect' was
    not fixed in FWaaS.

    This patch fixes it in the FWaaS to apply
    the Firewall rule in the router namespace
    for the North-South traffic.

    [1] https://review.openstack.org/#/c/283757

    Closes-Bug: #1715395
    Change-Id: Id8c902381f95e39bc13e3b3aeeeaa799c72f0dca

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-fwaas (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/554293

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-fwaas (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/554294

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron-fwaas 13.0.0.0b1

This issue was fixed in the openstack/neutron-fwaas 13.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-fwaas (stable/queens)

Reviewed: https://review.openstack.org/554293
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=5e967369e18a9a025a22a4b9e9fa8ba5707322e8
Submitter: Zuul
Branch: stable/queens

commit 5e967369e18a9a025a22a4b9e9fa8ba5707322e8
Author: Reedip <email address hidden>
Date: Thu Sep 7 05:36:11 2017 +0000

    DVR-FWaaS: Fix DVR FWaaS rules for fipnamespace

    FWaaS for DVR routers were only supported for
    North-South traffic. But with the recent change
    in the DVR router-info to handle FloatingIP's
    as mentioned in the reference patch[1], the
    dist_fip_count variable was removed and was
    replaced by rtr_fip_connect.

    So change in variable 'rtr_fip_connect' was
    not fixed in FWaaS.

    This patch fixes it in the FWaaS to apply
    the Firewall rule in the router namespace
    for the North-South traffic.

    [1] https://review.openstack.org/#/c/283757

    Closes-Bug: #1715395
    Change-Id: Id8c902381f95e39bc13e3b3aeeeaa799c72f0dca
    (cherry picked from commit 5706252c7947887e3c5b00f258dd847027497b97)

tags: added: in-stable-queens
tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-fwaas (stable/pike)

Reviewed: https://review.openstack.org/554294
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=e96a39f22f61aaf24e2d104170f1fd91d55212d7
Submitter: Zuul
Branch: stable/pike

commit e96a39f22f61aaf24e2d104170f1fd91d55212d7
Author: Reedip <email address hidden>
Date: Thu Sep 7 05:36:11 2017 +0000

    DVR-FWaaS: Fix DVR FWaaS rules for fipnamespace

    FWaaS for DVR routers were only supported for
    North-South traffic. But with the recent change
    in the DVR router-info to handle FloatingIP's
    as mentioned in the reference patch[1], the
    dist_fip_count variable was removed and was
    replaced by rtr_fip_connect.

    So change in variable 'rtr_fip_connect' was
    not fixed in FWaaS.

    This patch fixes it in the FWaaS to apply
    the Firewall rule in the router namespace
    for the North-South traffic.

    [1] https://review.openstack.org/#/c/283757

    Closes-Bug: #1715395
    Change-Id: Id8c902381f95e39bc13e3b3aeeeaa799c72f0dca
    (cherry picked from commit 5706252c7947887e3c5b00f258dd847027497b97)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron-fwaas 11.0.2

This issue was fixed in the openstack/neutron-fwaas 11.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron-fwaas 12.0.1

This issue was fixed in the openstack/neutron-fwaas 12.0.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.