Security vulnerability with SR-IOV ports

Bug #1797575 reported by Olivier De Jonckère
262
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Security Advisory
Won't Fix
Undecided
Unassigned
neutron
New
Undecided
Unassigned

Bug Description

As explain in http://www.mulix.org/pubs/misc/sriovsec-tr.pdf an attacker that has been assigned a VF of a NIC for its VM can block the network access for all the VMs using a VF of the same card by sending control flow PAUSE commands at the right interval.

The attack is described as hard to detect, easy to implement and absolutely efficient (throughput drops to 0).

A VF of a SR-IOV virtualized NIC can be assigned via pci aliases or with neutron ports.

I suppose with a VF assigned via a nova pci-passthrough these PAUSE commands would block the network. Would it be the case as well using the neutron port method ?

I don't have enough knowledge on neutron's functioning to see if these threats are serious or not, and I do not have the set up to test this myself.

Tags: sriov-pci-pt
Revision history for this message
Jeremy Stanley (fungi) wrote :

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Changed in ossa:
status: New → Incomplete
description: updated
Revision history for this message
Miguel Lavalle (minsel) wrote :

I discussed this CVE with Sean Mooney earlier today. I am adding him to this report, so he can give us his opinion

Revision history for this message
sean mooney (sean-k-mooney) wrote :

hi i unfortunately seam to have lost the analasys i did of this paper.

i will have to reconstruct it again but the TLDR that i remember is that
the paper does not provide a vialble workarount that neutron can implement
as it effectively would require a hardware change.

when considering if this is a viable exploit we also need to consider
hardware offloaded ovs.

for macvtap sriov we should be able to install an ntables filter to drop
all pause frames. for hardare offloaved ovs we should not need to if the implemented
flow control correctly but im not sure they have. we could add a drop rule to ovs also
for each vm port.

for vnic type direct phyical there is nothing neutron can do to filter out
pause frames however the data sheet for the nics they used state that only the PF can
send flow control messages which brings meaning vnic type direct should be if the PF is configred correctly which raises the question how they had configred the nics when doing there testing in the paper.

i will update this with future info related to ovs and the data sheet sections once i have a change to find the info again. it is unclear to me if this should be a neutron bug as the nics do have protections built into them to prevent this type of exploit which would indicate that this is either a hardware/firmware bug if present. assuming there is no firware bug its also possible that the nic was misconfigred or that the feature in the nic is not exposed/used by the dirver they used in testing.

so summary
- this could be a viable exploit.
- we may not be able to address this form openstack.
- it may require hardware/firmware change to address.
- the nic they used for testing has a feature to prevent this and its unclear if that was used.
- we may be able to implement a mitigation for vnic-type macvtap
- we cannot for vnic-type direct-physical
- for vnic-type direct we may be able to ensure the pf has flowcontol enable/disabeled from neutron
- this may also effect other backends that use sriov as a datapath such as hardware offloaded ovs.
- this would also affect nic passed through via nova falvor alias but those cannont be used with neutron so that is proably fine.

Revision history for this message
Jeremy Stanley (fungi) wrote :

In keeping with recent OpenStack vulnerability management policy changes, no report should remain under private embargo for more than 90 days. Because this report predates the change in policy, the deadline for public disclosure is being set to 90 days from today. If the report is not resolved within the next 90 days, it will revert to our public workflow as of 2020-05-27. Please see http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012721.html for further details.

description: updated
Revision history for this message
Jeremy Stanley (fungi) wrote :

It doesn't look like this report has seen any activity since my update two months ago, so consider this a friendly reminder:

The embargo for this report is due to expire one month from today, on May 27, and will be switched public on or shortly after that day if it is not already resolved sooner.

Thanks!

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

I'm adding Rodolfo to this bug as he is our main SR-IOV expert in the Neutron team.

Jeremy Stanley (fungi)
description: updated
Revision history for this message
Jeremy Stanley (fungi) wrote :

The embargo for this report has expired and is now lifted, so it's acceptable to discuss further in public.

description: updated
information type: Private Security → Public Security
tags: added: sriov-pci-pt
Revision history for this message
sean mooney (sean-k-mooney) wrote :

for what its worth i am still not conviced by the orginal paper and my assemetn was in general this is not an openstack bug if an explit does exsith although in limited case openstack might be able to add some mitigations.

sepecifcly macvtap sriov or ml2/ovs contoling hardware offlowaded ovs.
in both cases filtering could be applied to drop pause frames

for vnic_type direct we cant apply filtering but the nic they tested has a feature to prevent the exploit they reported allowign only the PF to send pause frames

if that was enabled then there would be no exploit so its still not clear that this was really a security bug for openstack vs a misconfitured test enviornment.

Revision history for this message
Jeremy Stanley (fungi) wrote :

Since nobody has disputed Sean's assertions in the nearly half a year since his comment #8 above, I'm going to assume the VMT no longer needs to track this and is unlikely to issue any security advisory about it, so am marking our advisory task Won't Fix.

Changed in ossa:
status: Incomplete → Won't Fix
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.