[RFE] Host Aware IPAM

There may be some synergy between this request and the routed networks work. I wonder if Calico could benefit from the current proposed solution for routed networks. Would it be crazy for Calico to implement enough of the segments API so that the interaction with Nova is the same as with an ML2 or other based plugin? (I suspect it might be)

One difference that I see between this and routed networks is that with the latter, subnets are hard-bound to segments. There really is no flexibility to break the boundary because of the underlying L2 semantics which must be respected. With Calico, I suspect the boundaries are soft. What I mean is that if one cluster of IPs is exhausted, you have the ability to assign an IP from another cluster and route it correctly. It sounds like you want to minimize the fragmentation of IP clusters to keep routing tables concise but you may or may not require absolute adherence to the boundaries. Is that right? Or, do you want the availability of IPs in a cluster to drive placement in Nova?

@Carl - Hi, thanks for your comments.

First, I should make clear that I'm mentoring Petra for this work, as part of the Outreachy program. Hence the overlap with the clustered IP allocation ideas that we've discussed a few times before. (As opposed to being a surprisingly detailed coincidence :-))

Secondly, I must admit that I'm behind on reviewing the latest revision of your spec, and it's indeed possible that it could now be pressed into service to deliver the clustered IP allocation that Calico wants. I plan to re-review the spec and form a detailed view on this in the next few days.

Even if that was the case, though, do you not think that it would also be interesting to explore using pluggable IPAM here? Pluggable IPAM is still a young technology, and in the short term I think an interesting area for a new developer to poke and contribute to; and in the long term I don't think it would be bad if we had two viable ways to achieve clustered IP allocation.

Your second paragraph above (apart from the last sentence) is exactly right. With Calico the boundaries are soft, because (1) we still want to allow arbitrary VM migration, and (2) we just need to cluster enough so that we can aggregate VM routes sufficiently, so that the complete required routing table will fit in an L3 fabric router.

A few more observations.

- Neil, the third point in your description is a prerequisite for the first. We have this need in the routed networks work too. There is a chance to collaborate on this.
- This work could be useful in the routed networks case when we begin to work floating IPs in to routed networks (A spec I will begin to write today). There may also be opportunity to collaborate here too.

A spec is proposed in neutron-specs: https://review.openstack.org/#/c/273868/

So, just discussed this in the drivers meeting. I wanted to first clear some things up.

1) The request is not for Nova scheduling to do anything different than it does today. The requirement to aggregate IPs around a host or a rack is a soft one. It is nice to group IP addresses into larger CIDRs so that the routing tables are smaller and not full of thousands of host routes. On the other hand, if one area runs out of IP addresses, a few odd host routes are acceptible. It would be pretty extreme to have nova boot fail because we don't want to add a host route when there is no technical reason we can't. Does that make sense?

2) In general, this depends on deferring IP address assignment until after port binding. This is a common requirement with the routed networks work going on in Neutron. You might be able to start this work without that by using some magic incantation that results in doing IPAM on an already bound port but that wouldn't really be generally useful.

I'd like the submitter to adjust the description to make these things clear. Let's get the spec fleshed out a bit. Given that we can have a port with binding done, what interfaces will need to be touched? I can help you on the spec and then we'll bring this back to the drivers.

Full log:

http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-02-18-22.01.log.html#l-113

Thanks Armando et al for discussing this in neutron-drivers! I appreciate your time on this. That said, I'm bemused that the IRC log has so many claims that this RFE wants scheduling-related things, when the RFE summary very clearly says that this is not the case:

"This work is being done as part of an Outreachy internship, and the last point cannot reasonably fit in that scope. Hence this RFE proposes just the first two points, as a useful and concrete step towards the eventual complete picture."

Thanks to Carl for pointing that out (a few times) during the discussion.

I now see, however, that the pluggable-IPAM-related aspect of this RFE was wrongly presented, and I apologize for that.

- Firstly, it should have been clearer that we (Petra Sargent and I) actually planned to work on this - as opposed to asking the Neutron community to implement it for us.

- Secondly, as our plan was to implement this as a pluggable IPAM driver, it arguably didn't actually need a Neutron core spec or RFE at all; instead we could have handled this as a networking-calico spec. (We certainly did want to involve and invite participation from the relevant Neutron experts such as John Belamaric and Pavel Bondar, but that would be equally possible for a networking-calico spec.)

Since raising this RFE, a couple further relevant things have happened.

1. Petra (and I) spent quite some time understanding the pluggable IPAM interface, and how we might implement a driver that could take the chosen compute host into account when allocating an IP for a VM. It has become clear that it isn't achievable to write that driver within the Outreachy timeframe, but that we could usefully contribute some devref documentation of our understanding.

2. I learnt of the Romana project's IPAM driver [1] which is doing similarly host-dependent IP allocation. On the one hand this confirms that it's possible (of course) to implement as a driver outside Neutron core, and also confirms much of the detail of how Petra and I had inferred that such a driver would need to be written. On the other hand it raises two questions about the current pluggable IPAM interface: (a) Given the IPAM driver complexity that is apparently needed for a pretty simple requirement (host-dependent IP allocation), have we got the pluggable IPAM interface right? (b) If there are already 2 backend projects wanting host-dependent IP allocation, should provision of binding:host_id be added to the core IPAM framework?

Right, so where does that leave us all, and this RFE?

Petra is shifting focus, for the remainder of her Outreachy time, to writing up what she has learnt about the pluggable IPAM interface, in the form of devref documentation. I hope that this will appear very soon as a Gerrit change (in neutron) for others to review and add to.

I have no immediate plan to focus on this more myself, beyond mentoring Petra. It's still an important point on the Calico roadmap, though, so likely that I will want to work on it again before very long. As regards possible implementations, I'm open-minded as to using pluggable IPAM, or the Segment objects in https://review.openstack.org/#/c/225384/, if those land i...

Woops, forgot to add reference:

[1] https://github.com/romana/networking-romana/blob/master/networking_romana/driver/ipam_romana.py

FYI here's the devref doc that Petra and I are now working on for pluggable IPAM: https://review.openstack.org/#/c/289460/

Status update:

http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-03-24-22.01.log.html#l-126

After some review and reading Neil's final comments, I'm going to close this. I'm about to crack IPAM for routed networks. With this request in mind, I'll be looking to evolve the IPAM interface so that it can take host information into consideration. I'll also be working on the deferred IP allocation.

I think there will be a way to do all of this which can accommodate both hard and soft boundaries for cidrs. I'll have some code up for review soon and will add Neil and Petra to the review.

In the meantime, I suggest we review and merge Petra's devref contribution: [1].

[1] https://review.openstack.org/#/c/289460/