I have a cluster of bare metal nodes managed by Icehouse OpenStack on CentOS 6.5. I'm using nova, ironic and neutron. The nodes each have a 1G management network interface and a 40G high speed data network interface. These interfaces are registered with their MAC addresses as ironic ports. There is a single flat neutron network for management and a single flat neutron network for the high speed network.
When booting a (baremetal) nova instance, if the instance's networks are specified by their network ID, there is no way (to my knowledge) to guarantee a mapping between interface and network. In 50% of cases, I will end up with a neutron port on the management network, with the MAC address of the node's 40G NIC, and vice versa. The implication of this is that DHCP address assignment fails, and the instance fails to provision with the following error message:
2014-12-23 05:03:20.877 227052 ERROR ironic.conductor.manager [-] Timeout reached when waiting callback for node 49f8b733-525d-44a2-ae19-596b19aa5d1a
My workaround is to manually create neutron ports for each network interface, specifying a MAC address and the appropriate neutron network. These ports are then referenced when booting an instance, instead of specifying a network ID. This workaround is not ideal, in particular because it only works when the nodes are pets (as opposed to cattle).
Based on my browsing of the nova source, I think the point of ill decision occurs at https://github.com/openstack/nova/blob/06c537fbe5bb4ac5a3012642c899df815872267c/nova/network/neutronv2/api.py#L270.
Simple minded, misguided half solution - tag the ironic ports with some metadata, such as the provider:physical_network tag of the neutron network to which they are attached. This can be used in nova to select an appropriate HW port for each of the instance's networks.
I have implemented the simple solution outlined in the description and it does indeed solve the problem. The implementation isn't perfect and would require some tweaking if it were to be submitted.