mariadb_recovery is prone to data loss
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| kolla-ansible |
Fix Released
|
Critical
|
Tudosoiu Marian | ||
Bug Description
While the mariadb_recovery playbook has never been the shining pinnacle of Galera recovery that it could be, it is in a pretty bad state right now.
The playbook in the past was "supply recovery host or it will recover the first host in the mariadb group (mariadb[0])". Now it is attempting to be a bit smarter and properly read the grastate.dat, but not accounting for non-graceful shutdowns.
A patch added prior to Newton reads the grastate.dat and tries to parse it for the highest seqno. The data-loss scenario is when you have shutdown a node gracefully, then some time passes, then your cluster crashes. This isn't uncommon and shouldn't be dismissed. When that happens the playbooks will choose the old, gracefully shutdown node and stomp the data on the rest of the nodes. This is all done without user interaction and is exceedingly dangerous since no backup is done either.
The proper recovery method that works automated a good portion of the time is as follows:
* Check if all mariadb nodes are stopped
* if not stopped then do not recover
* Check if any mariadb nodes have gvwstate.dat
* if gvwstate.dat found start *only* the nodes with gvwstate.dat without special options
* This is not a garaunteed recovery, but it is a safe action (no data loss can occur)
* Check if any mariadb nodes have grastate.dat
* If grastate.dat exists and has a seqno of -1 on any node, it is not safe to autorecover, abort
* if no -1 exists on any node, bootstrap the node with the highest seqno
This will cover a good chunk of the failure scenarios, including graceful shutdowns and full cluster outages without any risk of data. If it cannot automatically recover then the user should be *forced* to supply a bootstrap node on the command line based on whatever critera they want (sometimes guessing, but guessing should be done by the user, not Kolla-Ansible).
| Changed in kolla-ansible: | |
| importance: | Undecided → Critical |
| status: | New → Confirmed |
| Changed in kolla-ansible: | |
| milestone: | none → pike-2 |
| Changed in kolla-ansible: | |
| milestone: | pike-2 → pike-3 |
| Changed in kolla-ansible: | |
| milestone: | pike-3 → pike-rc1 |
| zongyimin (yanpeifei) wrote | #1 |
| Changed in kolla-ansible: | |
| milestone: | pike-rc1 → pike-rc2 |
| milestone: | pike-rc2 → queens-1 |
| Changed in kolla-ansible: | |
| milestone: | queens-2 → queens-3 |
| Changed in kolla-ansible: | |
| assignee: | nobody → Tudosoiu Marian (mtudosoiu) |
| OpenStack Infra (hudson-openstack) wrote Related fix proposed to kolla-ansible (master) | #2 |
| Changed in kolla-ansible: | |
| status: | Confirmed → Fix Committed |
| OpenStack Infra (hudson-openstack) wrote Change abandoned on kolla-ansible (master) | #3 |
| OpenStack Infra (hudson-openstack) wrote Related fix proposed to kolla-ansible (master) | #4 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to kolla-ansible (master) | #5 |
| OpenStack Infra (hudson-openstack) wrote | #6 |
| OpenStack Infra (hudson-openstack) wrote Change abandoned on kolla-ansible (master) | #7 |
| OpenStack Infra (hudson-openstack) wrote | #8 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to kolla-ansible (master) | #9 |
| Changed in kolla-ansible: | |
| status: | Fix Committed → Fix Released |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to kolla-ansible (stable/pike) | #10 |
| OpenStack Infra (hudson-openstack) wrote Related fix merged to kolla-ansible (master) | #11 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to kolla-ansible (stable/pike) | #12 |
| tags: | added: in-stable-pike |
| OpenStack Infra (hudson-openstack) wrote Related fix proposed to kolla-ansible (stable/pike) | #13 |
| OpenStack Infra (hudson-openstack) wrote Related fix merged to kolla-ansible (stable/pike) | #14 |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/kolla-ansible 6.0.0.0rc1 | #15 |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/kolla-ansible 5.0.2 | #16 |
I also meet it.