Pool Manager creates/deletes can go into an infinite loop
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Designate |
Fix Released
|
High
|
Tim Simmons |
Bug Description
The Pool Manager doesn't really handle the following situation:
1. Pool Manager tries to create/delete a zone. In BIND9, this results in a rndc addzone/delzone. In PowerDNS, this results in a database query.
2. Nameserver receives this communication and starts working on the operation.
3. The connection between Pool Manager and this nameserver drops, or times out, or network issues cause the returning message to be dropped. There are a litany of things that can cause this.
Result: The operation requested happened successfully, but Designate doesn't know about it.
Most of the time, the Pool Manager triggers the zone to ERROR, with a CREATE/DELETE action. Which, upon the next syncing cycle attempts to recreate/redelete the zone. This almost always fails, because the action has already been taken. If it does fail, this pattern repeats it self ad infinitum.
There are two ways to fix this, as I see it.
1. A backend can decide, when receiving a CREATE/DELETE whether it wants to check and see if that action has been completed already. Some backends make this easier than others. If that check shows the action has happened, the backend will return success to the Pool Manager, and life can go on.
2. As part of it's syncing process, the Pool Manager can go and inspect the nameserver via SOA query through MiniDNS before deciding to retry the CREATE/DELETE. This adds more overhead to syncing, and it doesn't seem to be an extremely common issue. But it's possible that this could be used for something more general.
Thoughts?
Changed in designate: | |
status: | In Progress → Fix Committed |
Changed in designate: | |
milestone: | none → kilo-rc1 |
Changed in designate: | |
importance: | Undecided → High |
Changed in designate: | |
status: | Fix Committed → Fix Released |
Changed in designate: | |
milestone: | kilo-rc1 → 2015.1.0 |
Fix proposed to branch: master /review. openstack. org/163901
Review: https:/