Comment 7 for bug 1649586
Revision history for this message
| Pawel Koniszewski (pawel-koniszewski) wrote | #7 |
© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
a60fb26
(Get the code!)
I did some investigation, here is what I found. IMHO it fails because of rabbit issues. Let's analyze it on an instance with ID af219925-
The first request of a VM happened at 03:01:15.451, the request ID is req-e0927ad1-
http://
This request ends up with a 500 error
http://
But few milliseconds before this error we can see another request to live migrate the same VM for the second time (which is not what live migration test should do):
http://
Look at the time - 03:02:15.446, it's more or less 1 minute after the first request. When calling urllib3 and creating retry mechanism we do not override 'total retries', we just create Retry object:
https:/
And here is the commit that changed tempest behavior, so that tempest creates PoolManager that timeouts exactly after 60 seconds:
https:/
So due to this timeout behavior and that we do not change retry logic while creating Retry object, urllib3 retries the live migration call, therefore we can see 409 CONFLICT error and failing test. This is also why we can see request ids that do not come from tempest, as those are requests generated by urllib3.
The real problem there is Rabbit and that it timeouts some requests and this is what we should fix. We can find the same timeouts in q-agt in gate-grenade-
http://