Scheduler Race Condition at high volume

A few commments here:

a) the availability-zone scheduling uses forced_host which skips the scheduler logic.

b) euca-run-instances -n actually uses a different code path than nova boot with a shell loop. I have not been able to reproduce your issue in this case.

c) The issue with the shell loop is reproducible and I have a patch which appears to fix it in the case of one scheduler. Multiple schedulers could be trickier.

Fix proposed to branch: master
Review: https://review.openstack.org/15215

After writing this I considered that availability-zone scheduling might just by pass scheduler logic since there is per force one choice.

I am definitely seeing this with the euca-run-instances -n case (I belive this is a single API request rather than the N api request the shell loop gives?).

Using the ChanceScheduler euca-run-instances -n 700 gives me 700 instance in active state in about 8min whit no over scheduling on compute nodes. I had about 800 unallocated VCPUs over 43 nodes when I ran this so it's a significant fraction of my available capacity.

Switching back to the FilterScheduler with the same command 230 instances got schedule on one node that only had 17 available.

I've seen similar behaviour with numbers as low as 100. (using packaged code not your patch)

Reviewed: https://review.openstack.org/15215
Committed: http://github.com/openstack/nova/commit/94560ab57d9fc23673f42017e6f2a78cb2b66b7a
Submitter: Jenkins
Branch: master

commit 94560ab57d9fc23673f42017e6f2a78cb2b66b7a
Author: Vishvananda Ishaya <email address hidden>
Date: Wed Oct 31 19:41:04 2012 -0700

    Eliminates simultaneous schedule race.

    Keeps host state in memory so multiple schedule attempts use the
    up-to-date values that may have been modified by another greenthread.

    Fixes bug 1073956

    Change-Id: I69fdd9b46bde6b7408c501c42a6ef3b6dd92bbc2

Status changed to 'Confirmed' because the bug affects multiple users.

I've verified that this affects Folsom as well and the patch does resolve the issue.

Hey Vish,

I'm still seeing the behavior happen when scheduling at least 52 instances nearly simultaneously. I am running with the patch mentioned above. We are running a single instance of the scheduler.

The scheduler placed the instances in the following manner:
compute-node07 : 4
compute-node09 : 3
compute-node10 : 3
compute-node11 : 3
compute-node22 : 18
compute-node26 : 13
compute-node36 : 4
compute-node37 : 4

all nodes have 96GB of RAM.

nova.conf:
compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
scheduler_available_filters=nova.scheduler.filters.standard_filters
# which filter class names to use for filtering hosts when not specified in the request.
#scheduler_default_filters=AvailabilityZoneFilter,RamFilter,ComputeFilter,CoreFilter,SameHostFilter,DifferentHostFilter,RetryFilter
scheduler_default_filters=RamFilter,ComputeFilter,RetryFilter
node_availability_zone=nova
default_schedule_zone=nova
compute_fill_first_cost_fn_weight=-1.0
scheduler_max_attempts=5

Here is a snippet from the nova-scheduler.log

2012-11-15 22:28:16 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on compute.compute-node22... from (pid=10327) cast /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:376
2012-11-15 22:28:16 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on compute.compute-node22... from (pid=10327) cast /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:376
2012-11-15 22:28:16 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on compute.compute-node22... from (pid=10327) cast /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:376
2012-11-15 22:28:16 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on compute.compute-node22... from (pid=10327) cast /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:376
2012-11-15 22:28:17 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on compute.compute-node22... from (pid=10327) cast /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:376
2012-11-15 22:28:18 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on compute.compute-node22... from (pid=10327) cast /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:376
2012-11-15 22:28:18 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on compute.compute-node22... from (pid=10327) cast /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:376
2012-11-15 22:28:18 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on compute.compute-node22... from (pid=10327) cast /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:376
2012-11-15 22:28:18 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on compute.compute-node22... from (pid=10327) cast /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:376
2012-11-15 22:28:18 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on compute.compute-node22... from (pid=10327) cast /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:376
2012-11-15 22:28:19 DEBUG nova.openstack.common.rpc.amqp [-] Maki...

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/16430

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/16587

Reviewed: https://review.openstack.org/16587
Committed: http://github.com/openstack/nova/commit/b874d21fcdd804ba8f92f06fbcf4b09c634995a4
Submitter: Jenkins
Branch: stable/folsom

commit b874d21fcdd804ba8f92f06fbcf4b09c634995a4
Author: Vishvananda Ishaya <email address hidden>
Date: Wed Oct 31 19:41:04 2012 -0700

    Eliminates simultaneous schedule race.

    Keeps host state in memory so multiple schedule attempts use the
    up-to-date values that may have been modified by another greenthread.

    Fixes bug 1073956

    Change-Id: I69fdd9b46bde6b7408c501c42a6ef3b6dd92bbc2
    (cherry picked from commit 94560ab57d9fc23673f42017e6f2a78cb2b66b7a)

Hello Jon, or anyone else affected,

Accepted nova into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nova/2012.2.1+stable-20121212-a99a802e-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!