neutron port API does not support atomicity

There is a related nova bug https://bugs.launchpad.net/nova/+bug/1493714

Can you provide more details on the use case? (e.g. why aren't the clients
just creating their own ports?)

On Mon, Sep 28, 2015 at 11:34 AM, Balazs Gibizer <
<email address hidden>> wrote:

> Public bug reported:
>
> Neutron port API offers an update method where the user of the API can say
> "I use this port" by setting the device_owner and device_id fields of the
> port. However the neutron API does not prevent port allocation race
> conditions.
> The API semantic is that a port is used if the device_id and the
> device_owner fields are set, and not used if they aren't. Now lets have
> two clients that both want to set the ownership of the port. Both clients
> first have to check if the port is free or not by checking the value of the
> device_owner and device_id fields of the port, then they have to set the
> those fields to express ownership.
> If the two clients act parallel it is pretty much possible that both
> clients see that the fields are empty and both issue the port update
> command. This can leads to race conditions between clients.
>
> ** Affects: neutron
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to
> neutron.
> Matching subscriptions: NeutronBugs
> https://bugs.launchpad.net/bugs/1500365
>
> Title:
> neutron port API does not support atomicity
>
> Status in neutron:
> New
>
> Bug description:
> Neutron port API offers an update method where the user of the API can
> say "I use this port" by setting the device_owner and device_id fields of
> the port. However the neutron API does not prevent port allocation race
> conditions.
> The API semantic is that a port is used if the device_id and the
> device_owner fields are set, and not used if they aren't. Now lets have
> two clients that both want to set the ownership of the port. Both clients
> first have to check if the port is free or not by checking the value of the
> device_owner and device_id fields of the port, then they have to set the
> those fields to express ownership.
> If the two clients act parallel it is pretty much possible that both
> clients see that the fields are empty and both issue the port update
> command. This can leads to race conditions between clients.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/neutron/+bug/1500365/+subscriptions
>

--
Kevin Benton

Hi Kevin,

There is single use case described in the https://bugs.launchpad.net/nova/+bug/1493714 where you can see that nova is the client using the neutron API and nova falls into a trap of a race conditions.

The whole problem can be dismissed by saying two clients should not try to use the same port at the same time.
But you can imagine that there are two independent clients in the system and both client thinks that the port belongs to its own domain. Then there is no way in the clients to detect that the other client thinks the same. Also there is no way to change the clients to fully avoid the race condition without changing the neutron API to have a port update method with and atomic test-and-set semantics.
You can say that it is a user error using the same port in two requests however even if it is true, there shall be a mechanism that detects the user error and rejects the request.

From API perspective, there is no restriction on port ownership within one tenant.
If tenant wants to change the ownership - it can do that. Also, there is no problems with atomicity, because API calls act quite atomically.
What you are looking for is transactional semantics where such client problem could be resolved, but I don't think neutron is going to provide such ability any time soon, and also I don't think it is on the map.

I actually thought that Kevin was going to look into something similar - a counter that would allow to trace resource generations, and that could then be used on API side to enforce specific generation for update operation, to allow one of the clients to fail in case of the race described.

We are aware of some of the limitations of the existing API. This will definitely require more attention across the board. Marking RFE, to raise the profile of this bug and seek input from members of the drivers team.

To be discussed.

Coming to think of it, why would anyone would boot two VMs with the same port ID? I understand that the race may indeed occur, but this feels a bit corner case; in theory we could reject update requests if the fields were already set, and explicitly introduce a clear operation that would reset the fields to NIL, but this feels awkward. I can't see any other way to provide atomicity across two (or more) rest calls; that would defeat the whole point of relying on something like REST, unless we want to build 2PC on top of REST.

Let's bring it up for discussion to see if there's any appetite to deal with this one way or another (server side). It looks like there might be a solution client side.

Based on discussion [1], if this becomes a burning issue for some other more realistic use case, we can consider introducing an atomic update of the device id and/or device owner field.

[1] http://eavesdrop.openstack.org/meetings/neutron_drivers/2015/neutron_drivers.2015-12-15-15.01.log.html