Novaclient retries after 504 timeout
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| python-novaclient |
Invalid
|
High
|
Unassigned | ||
Bug Description
This is strictly the fault of the underlying httplib2 library, but it causes bizarre behavior from python-novaclient, such as spawning 2X as many instances as were originally requested, and should be tracked upstream and fixed in python-novaclient once an appropriate upstream version exists.
We have nova-api configured behind HAProxy as a load balancer. HAProxy is currently configured to timeout requests after 50 seconds. Due to existing issues with Nova being really slow on database writes, if I request a large number of instances through the client (say, 80) then HAProxy will timeout the connection. However, the default behavior for httplib2 is to _IGNORE_ this timeout and the Connection: close it provides, and retry the request. When it does this, it re-establishes the connection from scratch so HAProxy sends the request to a new nova-api endpoint, which then also times out. In both requests, the request to spawn the instances eventually succeeds, and all 160 instances (2X the original request) come up fine.
It gets better. The value that controls the retry is hardcoded to 2 and not configurable in any released version of httplib2, although there appears to be a configurable fix in the trunk:
http://
See line 1227 or so.
HTTP POST request from python-novaclient (through Horizon dashboard) with HAProxy generated Timeout with Connection: close (that is summarily ignored by httplib2).
POST /v2/931f782cf7b
Host: proxy:8774
Content-Length: 179
x-auth-project-id: 931f782cf7b0461
accept-encoding: gzip, deflate
accept: application/json
x-auth-token: aa3bbff234ab4ab
user-agent: python-novaclient
content-type: application/json
{"server": {"name": "test_fail", "imageRef": "9d352713-
HTTP/1.0 504 Gateway Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
| Mark Gius (markgius) wrote | #1 |
| description: | updated |
| description: | updated |
| Changed in python-novaclient: | |
| importance: | Undecided → High |
| Andrew Laski (alaski) wrote | #2 |
| Changed in python-novaclient: | |
| status: | New → Triaged |
| Changed in python-novaclient: | |
| assignee: | nobody → Abhishek Lahiri (aviostack) |
| assignee: | Abhishek Lahiri (aviostack) → nobody |
| Abhishek Lahiri (aviostack) wrote | #3 |
| Davanum Srinivas (DIMS) (dims-v) wrote | #4 |
| Changed in python-novaclient: | |
| status: | Triaged → Invalid |
For anyone using HAProxy as we are, the HAProxy option "forceclose" can be used to rudely close the socket on httplib2, which avoids this bad behavior.