neutron

[OSSA 2014-014] neutron allows security group rules with invalid cidrs, resulting in broken iptables rules (breaking iptables-restore) (CVE-2014-0187)

Bug #1300785 reported by Christoph Thiel on 2014-04-01

260

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
OpenStack Security Advisory	Fix Released	High	Tristan Cacqueray
neutron	Fix Released	Undecided	Aaron Rosen	neutron 2014.2 "juno"
Havana	Fix Released	Undecided	Unassigned
Icehouse	Fix Released	Undecided	Unassigned

Bug Description

This bug is already reported in https://bugs.launchpad.net/neutron/+bug/1255338, but no security impact is discussed in that bug so far. We have been hitting the same issue in our cloud recently, and found that it is basically breaking quantum-plugin-openvswitch-agent / neutron-plugin-openvswitch-agent with errors like this:

security group was created with:

quantum security-group-rule-create default --direction egress --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix /32

    ERROR [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Error in agent event loop
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py", line 700, in rpc_loop
      sync = self.process_network_ports(port_info)
    File "/usr/lib/python2.7/dist-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py", line 655, in process_network_ports
      resync_a = self.treat_devices_added(port_info['added'])
    File "/usr/lib/python2.7/dist-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py", line 601, in treat_devices_added
      self.sg_agent.prepare_devices_filter(devices)
    File "/usr/lib/python2.7/dist-packages/quantum/agent/securitygroups_rpc.py", line 114, in prepare_devices_filter
      self.firewall.prepare_port_filter(device)
    File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
      self.gen.next()
    File "/usr/lib/python2.7/dist-packages/quantum/agent/firewall.py", line 107, in defer_apply
      self.filter_defer_apply_off()
    File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_firewall.py", line 284, in filter_defer_apply_off
      self.iptables.defer_apply_off()
    File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py", line 304, in defer_apply_off
      self._apply()
    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/lockutils.py", line 228, in inner
      retval = f(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py", line 340, in _apply
      root_helper=self.root_helper)
    File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py", line 61, in execute
      raise RuntimeError(m)
  RuntimeError:
  Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'iptables-restore']
  Exit code: 2
  Stdout: ''
  Stderr: "iptables-restore v1.4.12: host/network `' not found\nError occurred at line: 391\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n"

Our operations team is telling me, that just removing the broken role didn't help, but "restart quantum-plugin-openvswitch-agent" was needed on all affected nodes.

IMHO, this is an issue the vulnerability management team should consider. (We have been seeing this on stable/grizzly, but havana and upcoming icehouse are also affected.)

Related branches

lp:ubuntu/saucy-security/neutron

lp:ubuntu/trusty-security/neutron

CVE References

2014-0187

Thierry Carrez (ttx) on 2014-04-01

Changed in ossa:
status:	New → Incomplete

Revision history for this message

Thierry Carrez (ttx) wrote on 2014-04-02:

I think that's a valid DoS vector, due to its efficiency.

Not the first time an invalid iptables rule can be passed and wreck things -- maybe they should consider testing them in a sandbox before applying them, rather than try to sanitize them and then apply them blindly...

Changed in ossa:
importance:	Undecided → High
status:	Incomplete → Confirmed

Revision history for this message

Thierry Carrez (ttx) wrote on 2014-04-02:

Question: does it just break VMs that apply the rule, or the whole host ?

Revision history for this message

Christoph Thiel (cthiel-suse) wrote on 2014-04-02:

IIUC, it depends on your configuration if it affects the whole host, ore just a namespace / tenant. Someone with deeper understanding of the agent/linux/iptables_manager.py should give his view on the topic :)

Revision history for this message

Salvatore Orlando (salvatore-orlando) wrote on 2014-04-16:

The scope of the potential damage can vary as suggested by Cristoph.
It's not really a DoS, more an issue where security groups could be de-enforced without a user knowing it, since no information would be reflected in ports' statuses.

I agree about the security concern, but since the original bug is open, a security embargo is hardly enforceable now.

Revision history for this message

Aaron Rosen (arosen) wrote on 2014-04-16:

I agree this is a security concern. It prevents security group rules from being applied on a host after one installs that rule. I just reproduced this issue and tested this patch: https://review.openstack.org/#/c/59212/16 which fixes the issue. I think we should focus on getting that merged asap. I'm wondering if there should also be a security CVE released though so people are aware of the issue downstream.

Changed in neutron:
status:	New → Confirmed
assignee:	nobody → Aaron Rosen (arosen)

Revision history for this message

Thierry Carrez (ttx) wrote on 2014-04-18:

Even if the security aspect is not clearly described in the patch, we should assume that the patch is public and issue a CVE/OSSA on this ASAP. We can keep this bug private a bit more as we work out the details.

We'll need backports for this on stable/icehouse. Is Havana also affected ?

Changed in neutron:
status:	Confirmed → Fix Committed

Revision history for this message

Christoph Thiel (cthiel-suse) wrote on 2014-04-18:

Yes, havana is also affected. (As is grizzly, which is no longer an issue.)

I guess the OSSA draft is going to be put into this bug for review?

Revision history for this message

Tristan Cacqueray (tristan-cacqueray) wrote on 2014-04-18:

@Christoph Yes, here it is.

I have a question, is the iptables-restore issue will also prevent former rules from being re-applied or will it break at the
time the invalid rule is actually being applied ?
In such a case, the impact desc would need a note about that in case of a host reboot.

Impact description draft #1:

Title: Neutron security group DoS through invalid CIDR
Reporter: Stephen Ma (HP)
Products: Neutron
Versions: 2013.1 to 2013.2.3, and 2014.1 versions

Description:
Stephen Ma from Hewlett Packard reported a vulnerability in Neutron security group. By creating a security group rule with an invalid CIDR, an authenticated user may break openvswitch-agent process, preventing further rules from being applied on the host. Note: removal of the faulty rule is not enough, the openvswitch-agent must be restarted.
All Neutron setups are affected.

Changed in ossa:
assignee:	nobody → Tristan Cacqueray (tristan-cacqueray)

Revision history for this message

Thierry Carrez (ttx) wrote on 2014-04-18:

Comments:
* would just say "Versions: 2013.1 to 2013.2.3, and 2014.1" (since 2014.1 is the only "2014.1 version" that will be affected)
* reported a vulnerability in Neutron security *groups*
* I would say "bypass" rather than DoS, since it allows to evade security groups rather than deny service.

Revision history for this message

Christoph Thiel (cthiel-suse) wrote on 2014-04-18:

#10

I understand Stephan Ma first reported it, but didn't point out security issue. I wouldn't mind being mentioned as well, i.e. Christoph Thiel (Deutsche Telekom) ;)

In addition to that I would recommend noting, that only Neutron setups are affected that are using the OpenVswitch.

Revision history for this message

Tristan Cacqueray (tristan-cacqueray) wrote on 2014-04-18:

#11

Thanks for your comments!

Impact description draft #2:

Title: Neutron security groups bypass through invalid CIDR
Reporters: Stephen Ma (HP) and Christoph Thiel (Deutsche Telekom)
Products: Neutron
Versions: 2013.1 to 2013.2.3, and 2014.1

Description:
Stephen Ma from Hewlett Packard and Christoph Thiel from Deutsche Telekom reported a vulnerability in Neutron security groups. By creating a security group rule with an invalid CIDR, an authenticated user may break openvswitch-agent process, preventing further rules from being applied on the host. Note: removal of the faulty rule is not enough, the openvswitch-agent must be restarted. All Neutron setups using Open vSwitch are affected.

Revision history for this message

Thierry Carrez (ttx) wrote on 2014-04-22:

#12

Looks good to me

Tristan Cacqueray (tristan-cacqueray) on 2014-04-22

summary:

neutron allows security group rules with invalid cidrs, resulting in
- broken iptables rules (breaking iptables-restore)
+ broken iptables rules (breaking iptables-restore) (CVE-2014-0187)

Thierry Carrez (ttx) on 2014-04-22

Changed in ossa:
status:	Confirmed → In Progress

Tristan Cacqueray (tristan-cacqueray) on 2014-04-22

summary:	- neutron allows security group rules with invalid cidrs, resulting in - broken iptables rules (breaking iptables-restore) (CVE-2014-0187) + [OSSA 2014-014] neutron allows security group rules with invalid cidrs, + resulting in broken iptables rules (breaking iptables-restore) + (CVE-2014-0187)
information type:	Private Security → Public Security

Thierry Carrez (ttx) on 2014-04-22

Changed in ossa:
status:	In Progress → Fix Committed

Revision history for this message

Thierry Carrez (ttx) wrote on 2014-04-25:

#13

Still needs to be merged in stable/havana: https://review.openstack.org/#/c/88057/

Revision history for this message

Jeremy Stanley (fungi) wrote on 2014-05-06:

#14

stable/havana change merged April 30 at 02:36 UTC.

Thierry Carrez (ttx) on 2014-05-06

Changed in ossa:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-06-12

Changed in neutron:
milestone:	none → juno-1
status:	Fix Committed → Fix Released

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2014-06-18:

#15

FYI, I tried to backport the havana patch to Ubuntu 13.10 and have 3 new test failures as can be see in:
https://launchpadlibrarian.net/177272442/buildlog_ubuntu-saucy-i386.neutron_1%3A2013.2.3-0ubuntu1.3_UPLOADING.txt.gz

as opposed to:
https://launchpadlibrarian.net/174614850/buildlog_ubuntu-saucy-i386.neutron_1%3A2013.2.3-0ubuntu1.1_UPLOADING.txt.gz

Specifically:
FAIL: neutron.tests.unit.linuxbridge.test_lb_security_group.TestLinuxBridgeSecurityGroups.test_create_security_group_rule_invalid_ip_prefix
FAIL: neutron.tests.unit.linuxbridge.test_lb_security_group.TestLinuxBridgeSecurityGroupsXML.test_create_security_group_rule_invalid_ip_prefix
FAIL: neutron.tests.unit.midonet.test_midonet_plugin.TestMidonetSecurityGroup.test_create_security_group_rule_invalid_ip_prefix

I applied patchset 10 from https://review.openstack.org/#/c/88057/. Am I missing another patchset?

(There are some different failures in neutron.tests.unit.ml2.drivers.test_cisco_mech.* but I think this may be a problem in the testsuite.)

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2014-06-18:

#16

Nevermind, I just now found https://review.openstack.org/#/c/88058/.

Thierry Carrez (ttx) on 2014-10-16

Changed in neutron:
milestone:	juno-1 → 2014.2

Report a bug

This report contains Public Security information

Everyone can see this security related information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.