tempest

Request rate too high during test_security_group_rules running

Bug #1087298 reported by cwang16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tempest
Fix Released
Undecided
Unassigned

Bug Description

[Description]
The request rate is too high when running test_security_group_rules.py, which leads the request failure and test error.

[Error Log]
======================================================================
ERROR: Positive test: Creation of Security Group rule
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/april/latest/latest/tempest/tests/compute/security_groups/test_security_group_rules.py", line 97, in test_security_group_rules_create_with_optional_arguments
    group_rule_id = rule['id']
UnboundLocalError: local variable 'rule' referenced before assignment
-------------------- >> begin captured stdout << ---------------------
base_urlhttp://10.239.80.138:8774/v2/e66a888570dd4842a84c537a19cf1623

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
tempest.common.rest_client: ERROR: Request URL: http://10.239.80.138:8774/v2/e66a888570dd4842a84c537a19cf1623/os-security-groups
tempest.common.rest_client: ERROR: Request Body: <?xml version="1.0" encoding="UTF-8"?>
<security_group name="securitygroup-292691"><description >description-67789</description></security_group>
tempest.common.rest_client: ERROR: Response Headers: {'retry-after': '4', 'date': 'Thu, 06 Dec 2012 15:11:54 GMT', 'content-length': '217', 'status': '413', 'content-type': 'text/html; charset=UTF-8'}
tempest.common.rest_client: ERROR: Response Body: {'message': 'This request was rate-limited.', 'code': '413', 'details': 'Only 10 POST request(s) can be made to * every minute.'}
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 16 tests in 3.089s

[Steps to Duplicate]
Run under the /tempest/tests/compute/security_groups folder:
nosetests -v test_security_group_rules

Revision history for this message
cwang16 (chun-wang) wrote :

Runing test_security_groups.py also got this failure.

Revision history for this message
Jay Pipes (jaypipes) wrote :

You need to turn off rate-limiting in your openstack configuration (nova.conf) when running Tempest... we don't have any control over the rate limits in Tempest :)

Revision history for this message
cwang16 (chun-wang) wrote :

How about the real deployment environment which need to configure the rate limiation? Why the script cannot add some sleep time to avoid the default request rate limitation error?

Revision history for this message
cwang16 (chun-wang) wrote :

Suppose adding a re-try mechanism when geting this error is also working.

Revision history for this message
Jay Pipes (jaypipes) wrote :

We added the ability for the tempest test cases to create their own users/tenants to get around the problem with rate-limiting. Do you have allow_tenant_isolation in your tempest.conf?

Revision history for this message
cwang16 (chun-wang) wrote :

allow_tenant_isolation is usually set to false as default in our environment, since when allow_tenant_isolation = true, the teardown function not working issue is much more serious than when it's disabled. There is no way to batch execute cases in numbers, since the garbage instances/image snapshot and security groups are all easy exceed quota and leads to large number of following error cases.

Another question is here...how can I know for which case I should enable allow_tenant_isolation? I found behaviors of some scripts are not consistent when the setting changes.

Revision history for this message
Jay Pipes (jaypipes) wrote :

OK, so two things:

1) the garbage instances/snapshots/secgroups should all be cleaned up at the end of tests -- if they aren't, that's a bug and we will fix it immediately.

2) Using tenant isolation, you shouldn't run into quota issues because quotas are per tenant

3) For your final question, you don't enable tenant isolation per test case -- it's a global configuration setting. Which specific behaviours have you found to be inconsistent? We can file bugs on those.

All the best,
-jay

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tempest (master)

Fix proposed to branch: master
Review: https://review.openstack.org/17632

Changed in tempest:
assignee: nobody → Sean Dague (sdague-b)
status: New → In Progress
Sean Dague (sdague)
Changed in tempest:
assignee: Sean Dague (sdague-b) → nobody
status: In Progress → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tempest (master)

Reviewed: https://review.openstack.org/17632
Committed: http://github.com/openstack/tempest/commit/1c6722e447aa5b9367388f1a7845b5982a0b9ee6
Submitter: Jenkins
Branch: master

commit 1c6722e447aa5b9367388f1a7845b5982a0b9ee6
Author: Sean Dague <email address hidden>
Date: Thu Dec 6 15:37:03 2012 -0500

    early failures would prevent cleanup

    because we are using finally: to do cleanup, we are cleaning
    up variables which may never have been allocated, thus throwing
    and additional exception and leaving state around after our
    shutdown.

    Discovered by bug #1087298 however this is not a fix for that
    bug

    Change-Id: I395301e619cc290367aa0ab8fb02d729343f1432

Changed in tempest:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.