Comment 9 for bug 1781710

Revision history for this message
Matt Riedemann (mriedem) wrote :

(2:14:30 PM) mriedem: i think this is the killer https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py#L98
(2:15:54 PM) mriedem: when we're doing multi-create, we have 1 request spec per instance, so in this failing test we have 2
(2:15:55 PM) sean-k-mooney: checking if the instance uuid is already in the current host.
(2:16:10 PM) mriedem: but the conductor just randomly picks the first request spec from the list when passing it to the scheduler,
(2:16:15 PM) mriedem: and that request_spec has a random instance_uuid on it
(2:16:41 PM) mriedem: so if i've got hosts A and B and instances X and Y, and using reqspec with instance X,
(2:16:52 PM) mriedem: then go through host filtering and select host A for instance X,
(2:17:11 PM) mriedem: when i go through host filtering for instance Y and host A, it will say: if spec_obj.instance_uuid in host_state.instances.keys():
(2:17:13 PM) sean-k-mooney: oh realy should we not be looping over the requests specs in the multicreate request
(2:17:23 PM) mriedem: if X in A.instances(): return True
(2:17:43 PM) mriedem: the problem is when this code was added https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py#L98 it assumes resize, which is a single instance,
(2:18:05 PM) mriedem: and it worked because before https://review.openstack.org/#/c/571166/ the filter didn't look at host_state.instances within a multi-create create
(2:18:16 PM) mriedem: s/look at/depend on/
(2:19:10 PM) mriedem: i'm going to see if i can make this a predictable order through the scheduler to tickle the failure
(2:23:54 PM) sean-k-mooney: ya so that looks wrong https://github.com/openstack/nova/blob/039f7e055ee2b47e96be4e86d6f48d9ce469c123/nova/conductor/manager.py#L1191-L1194
(2:24:14 PM) sean-k-mooney: its using the request spec from the first instance only
(2:24:30 PM) mriedem: well, it's not wrong, it's just very brittle assumptions in the anti-affinity filter that didn't used to be a problem