[SRU] Active scrub blocks upmap balancer
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu Cloud Archive |
Invalid
|
Medium
|
Unassigned | ||
Ussuri |
Fix Released
|
Medium
|
Unassigned | ||
ceph (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Bionic |
Invalid
|
Medium
|
Unassigned | ||
Focal |
Fix Released
|
Medium
|
Unassigned | ||
Groovy |
Fix Released
|
Medium
|
Unassigned | ||
Hirsute |
Fix Released
|
Medium
|
Unassigned |
Bug Description
[Impact]
When scrubs are in progress, balancer stop due to the bug [0]. And shows:
<timestamp> calc_pg_upmaps abort due to max <= 0
in the logs.
Typically when deep-scrub is done in maintenance windows and can take few hours. If balancing is paused for the duration, it can affect client I/O performance later when balacing starts happening after deep-scrub is done.
This bug was introduced in Octopus. We need to backport upstream bug [0] to just Octopus. It's been fixed in upstream master branch [1].
[Test Case]
In an Octopus Ceph cluster that has some data (large enough to be able to notice balancing), take down one or more OSDs to introduce "unbalanced" objects.
Make sure Ceph balancer module is enabled and active (which should be the default case in Octopus).
Perform some I/O so that data goes to the rest of the OSDs.
Then start deep-scrubbing and re-add the previously taken down so balancing start to happen.
[Regression potential]
Low potential. This is a bug fix of a previously correct code.
If anything goes wrong, the balancer module might not function properly and thus
leaving the cluster unbalanced and potentially requiring manual balancing.
[Other Info]
It's been accepted upstream and backported to Octopus. Ref [0] and [1].
[0] https:/
[1] https:/
For Hirsute, James is working on a snapshot of Pacific and that should include the fix for this.
description: | updated |
tags: | added: seg sts |
Changed in cloud-archive: | |
status: | New → In Progress |
summary: |
- Active scrub blocks upmap balancer + [SRU] Active scrub blocks upmap balancer |
description: | updated |
no longer affects: | ceph (Ubuntu Hirsute) |
no longer affects: | ceph (Ubuntu Groovy) |
no longer affects: | ceph (Ubuntu Focal) |
affects: | ceph (Ubuntu) → focal (Ubuntu) |
affects: | focal (Ubuntu) → ceph (Ubuntu) |
tags: | added: sts-sru-needed |
Changed in cloud-archive: | |
importance: | Undecided → Medium |
Changed in ceph (Ubuntu Bionic): | |
importance: | Undecided → Medium |
Changed in ceph (Ubuntu Focal): | |
importance: | Undecided → Medium |
Changed in ceph (Ubuntu Groovy): | |
importance: | Undecided → Medium |
Changed in ceph (Ubuntu Hirsute): | |
importance: | Undecided → Medium |
description: | updated |
Changed in ceph (Ubuntu Hirsute): | |
status: | New → Triaged |
Changed in ceph (Ubuntu Groovy): | |
status: | New → Triaged |
Changed in ceph (Ubuntu Focal): | |
status: | New → Triaged |
Changed in cloud-archive: | |
status: | In Progress → Invalid |
Attaching debdiff for Focal/Octopus.