Creating server did not fail when exceeded Quantum quota limit

I using the released Grizzly bits 2013.1.1 with one controller node, one quantum node, and 4 compute nodes.

I encountered the issue where the Quantum port quota was exceeded, but the nova create instance did not return error condition for status. It appear to be bouncing the request to the 4 compute node where 3 of the nodes responded with quantum quota exceeded. Shouldn't the create request fail immediately on the first failure? Or from the log file, shouldn't it fail after 3 attempt?

Portion of my test log showing create instances:

Date Time Test server name task status
========== ======= ========================== ========= =====
[2013-06-17][13:07:09][A:0,E:0] Testing server testserver-20130617-130707-25 - networking, BUILD
[2013-06-17][13:07:15][A:0,E:0] Testing server testserver-20130617-130707-25 - None, ACTIVE
[2013-06-17][13:07:16][A:0,E:0] Waiting for server testserver-20130617-130715-26 with ID f6fb6968-82e7-4b36-87cc-c18eba189f71 to get spawned
[2013-06-17][13:07:18][A:0,E:0] Testing server testserver-20130617-130715-26 - networking, BUILD
[2013-06-17][13:07:24][A:0,E:0] Testing server testserver-20130617-130715-26 - None, ACTIVE
[2013-06-17][13:07:25][A:0,E:0] Waiting for server testserver-20130617-130724-27 with ID 4ed9a635-6fc4-4d4b-9a64-8dfd4574fddd to get spawned
[2013-06-17][13:07:26][A:0,E:0] Testing server testserver-20130617-130724-27 - networking, BUILD
[2013-06-17][13:07:32][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:07:37][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:07:43][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:07:49][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:07:55][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:08:01][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:08:07][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:08:13][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:08:19][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:08:25][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:08:31][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:08:36][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:08:42][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:08:48][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:08:54][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD
[2013-06-17][13:09:00][A:0,E:0] Testing server testserver-20130617-130724-27 - scheduling, BUILD

nova-scheduler.log
2013-06-17 13:07:26.287 ERROR nova.scheduler.filter_scheduler [req-816e9730-fe41-493a-93c5-ca79237705eb 6891ad89906646e4934ae6014eb07521 29ca96af08984e9288ec3f98a743197a] [instance: 4ed9a635-6fc4-4d4b-9a64-8dfd4574fddd] Error from last host: cld5b3 (node cld5b3.casl.adapps.hp.com): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 831, in _run_instance\n requested_networks, macs, security_groups)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1075, in _allocate_network\n instance=instance)\n', u' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1071, in _allocate_network\n security_groups=security_groups)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 46, in wrapper\n res = f(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 281, in allocate_for_instance\n \'exception\': ex})\n', u' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 264, in allocate_for_instance\n quantum.create_port(port_req_body)[\'port\'][\'id\'])\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params\n ret = self.function(instance, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 269, in create_port\n return self.post(self.ports_path, body=body)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 987, in post\n headers=headers, params=params)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request\n self._handle_fault_response(status_code, replybody)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response\n exception_handler_v20(status_code, des_error_body)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 80, in exception_handler_v20\n message=error_dict)\n', u"QuantumClientException: Quota exceeded for resources: ['port']\n"]
2013-06-17 13:07:28.584 ERROR nova.scheduler.filter_scheduler [req-816e9730-fe41-493a-93c5-ca79237705eb 6891ad89906646e4934ae6014eb07521 29ca96af08984e9288ec3f98a743197a] [instance: 4ed9a635-6fc4-4d4b-9a64-8dfd4574fddd] Error from last host: cld5b5 (node cld5b5.casl.adapps.hp.com): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 831, in _run_instance\n requested_networks, macs, security_groups)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1075, in _allocate_network\n instance=instance)\n', u' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1071, in _allocate_network\n security_groups=security_groups)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 46, in wrapper\n res = f(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 281, in allocate_for_instance\n \'exception\': ex})\n', u' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 264, in allocate_for_instance\n quantum.create_port(port_req_body)[\'port\'][\'id\'])\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params\n ret = self.function(instance, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 269, in create_port\n return self.post(self.ports_path, body=body)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 987, in post\n headers=headers, params=params)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request\n self._handle_fault_response(status_code, replybody)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response\n exception_handler_v20(status_code, des_error_body)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 80, in exception_handler_v20\n message=error_dict)\n', u"QuantumClientException: Quota exceeded for resources: ['port']\n"]
2013-06-17 13:07:30.398 ERROR nova.scheduler.filter_scheduler [req-816e9730-fe41-493a-93c5-ca79237705eb 6891ad89906646e4934ae6014eb07521 29ca96af08984e9288ec3f98a743197a] [instance: 4ed9a635-6fc4-4d4b-9a64-8dfd4574fddd] Error from last host: cld5b4 (node cld5b4.casl.adapps.hp.com): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 831, in _run_instance\n requested_networks, macs, security_groups)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1075, in _allocate_network\n instance=instance)\n', u' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1071, in _allocate_network\n security_groups=security_groups)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 46, in wrapper\n res = f(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 281, in allocate_for_instance\n \'exception\': ex})\n', u' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 264, in allocate_for_instance\n quantum.create_port(port_req_body)[\'port\'][\'id\'])\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params\n ret = self.function(instance, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 269, in create_port\n return self.post(self.ports_path, body=body)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 987, in post\n headers=headers, params=params)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request\n self._handle_fault_response(status_code, replybody)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response\n exception_handler_v20(status_code, des_error_body)\n', u' File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 80, in exception_handler_v20\n message=error_dict)\n', u"QuantumClientException: Quota exceeded for resources: ['port']\n"]
2013-06-17 13:07:30.399 WARNING nova.scheduler.manager [req-816e9730-fe41-493a-93c5-ca79237705eb 6891ad89906646e4934ae6014eb07521 29ca96af08984e9288ec3f98a743197a] Failed to schedule_run_instance: No valid host was found. Exceeded max scheduling attempts 3 for instance 4ed9a635-6fc4-4d4b-9a64-8dfd4574fddd