object auditors don't finish

Clay, what do you think of this 2-year-old bug report?

This is exactly the kind of *important* stuff that's getting buried in our backlog.

Object-auditors not finishing is an issue for every cluster of size.

Ideally someone other than the reported review the description and make sure the issue well stated and understood - then mark it as confirmed - probably with a low/wishlist priority.

[Expired for OpenStack Object Storage (swift) because there has been no activity for 60 days.]

In HP Public cloud we do graph the object auditor cycle times (min, avg and max).

In our us-west data center, the average object auditor cycle time is 11.5 days. For some curious reason the max cycle time is about twice that at 23 days. I’m not sure why that is since the average disk usage is about 56% used and the max disk usage is about 62% i.e. not that much above the average.

The object servers have 12 x 2TB disks and on average they are 56% used. So it is taking on avg 1 day for the object auditor to work through a 2TB drive at 56% used.

Of course these are oldish systems with relatively small drives. When using larger drives say 8TB at the same usage level of 56% , the object auditor could take in the order of 8 days to process the drive.

Logging a message on completing a device, and picking up from there on restart would be a big improvement.

If it's taking multiple days to process a drive, then remembering the last partition starts to sound like a good idea.

Chiming in here since this has been a problem at SoftLayer/IBM as well and we're very interested in similar improvements here -- a few folks have been actively looking at this a lot lately.

So logging a message after finishing audit of a device would be helpful, as well as having it remember which partition it left off on in the event of a restart.

Two questions though --

(a) If the auditor starts back up and its last checked partition is no longer there, do we care at that point whether it starts back over from the beginning versus continuing on to the next partition that would fall after its last checked partition? Similar to how markers work in a container listing... I'm thinking about this in the scenario where a node needs to go offline for maintenance and a ring change occurs while it's napping.

(b) Another solution brought up internally -- perhaps one auditor process per disk might be a better approach to this?

Particularly in the case where a data node has a lot of disks and the auditor consistently gets restarted sooner than it can complete a full cycle across the node, the last few disks may never get audited at all. At least in this scenario they have just as good a chance as the first disk, and the audit times for the node as a whole would seem to be a lot more predictable.

This also seems more sound in the scenario where you have disks under heavy read/write cycles due to rebalance replication -- if the auditor gets slowed down moving through those disks on a ring change, the successive disks can still be audited sooner.

@BC - great points!

I think for this to make sense you have to say "here's what I observed, and the order I plan to attack, here's a pointer to the last thing I finished"

not so sure about doing a design discussion on the bug... but maybe...

in the root of each device, each datadir/auditor-type get's it's own <type>_auditor_progress.json

which is in the form:

{
    part_list: [746, 2978, 10, ...]
    start_time: <timestamp>,
    last_update: <timestamp>,
    last_part: 678,
}

After every part the worker for that devices processes, we read the file, change last_part to be the one we just finished and write down the new last_update and last_part to indicate when we finished that part.

on start up, and before every part, we read the file, find the last_part in the part_list and start working on the next one

... if we ever find that the last_part in the *is* the last part in the part_list - we make big noisy log lines and recon dumps about finishing a device audit then do a listdir on the datadir to get a list of parts and start over again.

Fix proposed to branch: master
Review: https://review.openstack.org/279440

Reviewed: https://review.openstack.org/279440
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=fd86d5a95d73714365c07cb36bfd1404306142a7
Submitter: Jenkins
Branch: master

commit fd86d5a95d73714365c07cb36bfd1404306142a7
Author: Christian Schwede <email address hidden>
Date: Mon Feb 15 19:17:01 2016 +0000

    Skip already checked partitions when auditing objects after a restart

    The object auditor will save a short status file on each device, containing a
    list of remaining partitions for auditing. If the auditor is restarted, it will
    only audit partitions not yet checked. If all partitions on the current device
    have been checked, it will simply skip this device. Once all partitions on all
    disks are successfully audited, all status files are removed.

    Closes-Bug: #1183656

    Change-Id: Icf1d920d0942ce48f1d3d374ea4d63dbc29ea464

This issue was fixed in the openstack/swift 2.7.0 release.

Fix proposed to branch: feature/crypto
Review: https://review.openstack.org/299944

Reviewed: https://review.openstack.org/299944
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=77c181161a029ba8cea5d1ce65f979fe4b23ff37
Submitter: Jenkins
Branch: feature/crypto

commit 59bbe27fb0a40236108f09c9b3349e8faef0a95c
Author: Nguyen Hung Phuong <email address hidden>
Date: Wed Mar 30 11:07:46 2016 +0700

    Fix typos in Swift files

    Change-Id: I34e0c9a888127704ac1910e73ddd14e27ebade13

commit 7be55acf1bc4aa07d81b30fd93e144700889898d
Author: Tim Burke <email address hidden>
Date: Thu Feb 11 16:00:38 2016 -0800

    Simplify policy-name validation slightly

    _validate_policy_name always either returns True or raises an exception.
    Simplify it to just being a callable that may raise an exception.

    Also, move the check for blank/None names into _validate_policy_name, so
    it will be applied in more cases.

    Change-Id: I7832a0c9c895cd75ba4c6d0e8b5568a3c8a0ea25

commit 5902015fa8495ec0ef3c1ab92ae9a34c5bda4334
Author: OpenStack Proposal Bot <email address hidden>
Date: Sat Mar 26 06:35:18 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I3b5d401649fa3dea6dc43654516f7075bb06ee0d

commit 2f7d0f4a2ad2da7e6a35e5b054a47a2fafe5ed01
Author: Anh Tran <email address hidden>
Date: Fri Mar 25 11:44:26 2016 +0700

    Removing some redundant words

    This patch removes some redundant words.

    Change-Id: Ia79717664b06ed9a41c3c5dcf1a25e9e49e21cf2

commit 925546ae8a211b50cf7fad6634d47fd1dbfeb58e
Author: OpenStack Proposal Bot <email address hidden>
Date: Fri Mar 25 06:36:40 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I6ba2f35913e6ae83607b5e268645432d455d587c

commit 3407d737c705a7afedeed0159588ab4433a601f3
Author: David Liu <email address hidden>
Date: Thu Mar 24 16:08:19 2016 +0800

    Handle tempurl Content-Disposition header missing from HEAD

    Content-Disposition headers should make no difference between
    GET and HEAD according to HTTP rfc.

    Closes-Bug: #1539805

    Change-Id: Ifa41a7cda2f321eb8e36420ede7912ed0a549712

commit 2f24fb9683a57b67348d65864d5af8c3a03dee67
Author: Alistair Coles <email address hidden>
Date: Wed Mar 23 20:49:50 2016 +0000

    Check marker params in SimpleClient full listing requests

    Follow up for change [1] to add some assertions to check that
    marker param is included in sequential GET requests sent during
    a full listing.

    Extract multiple FakeConn class definitions to single class at
    module level and share between all classes.

    Also, explicitly unpack the return values from base request calls
    made in the full listing section of base_request, and explicitly
    return a list to make more consistent with rest of the method.

    [1] Change-Id: I6892390d72f70f1bc519b482d4f72603e1570163

    Change-Id: Iad038709f46364b8324d25ac79be4317add79df...

Fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/323599

Reviewed: https://review.openstack.org/323599
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0330478b70d0a699a0f9c21ef87c7e639d92564b
Submitter: Jenkins
Branch: feature/hummingbird

commit 5fe392b562de3baed080704df433fb392cb4fb31
Author: Ondřej Nový <email address hidden>
Date: Tue May 31 16:25:50 2016 +0200

    Fixed typo

    Change-Id: I7a35c0076360c7a23cf405189828d3c252ec6708

commit b52eccb3b1ea0591f0040587228d3705b5d3f68d
Author: Clay Gerrard <email address hidden>
Date: Wed May 25 11:21:25 2016 -0700

    Clarify overload best practices in admin guide

    Change-Id: Ib7c08bdeab6374771bb8e2b05053e7e16973524d

commit f1fd50723bb84c4941e949895576733f6eb67793
Author: Christian Schwede <email address hidden>
Date: Wed May 25 09:53:31 2016 +0200

    Add dispersion --verbose example to admin guide

    Change-Id: I5f9cacedde2a329332ccf744800b6f2453e8b28e

commit b3ab715c055283ccfea9a504d6da20741d82e7ad
Author: Matthew Oliver <email address hidden>
Date: Wed May 25 14:35:54 2016 +1000

    Add ring-builder dispersion command to admin guide

    This change updates the admin guide to point out the dispersion command
    in swift-ring-builder and mentions the dispersion verbose table to make
    it more obvious to operators.

    Change-Id: I72b4c8b2d718e6063de0fdabbaf4f2b73694e0a4

commit fb7a8e9ab7596a36a6992a3a8f8c6d005a2c2829
Author: Tim Burke <email address hidden>
Date: Tue May 24 13:37:58 2016 -0700

    Add links to mitaka install guides

    Change-Id: I62331923751c521daded4468b5cc5f03655226bc

commit e09c4ee7800e82aa09ca2f6ae375420b766182a4
Author: Tim Burke <email address hidden>
Date: Fri Apr 29 12:12:00 2016 -0500

    Allow concurrent bulk deletes

    Before, server-side deletes of static large objects could take a long
    time to complete since the proxy would wait for a response to each
    segment DELETE before starting the next DELETE request.

    Now, operators can configure a concurrency factor for the slo and bulk
    middlewares to allow up to N concurrent DELETE requests. By default, two
    DELETE requests will be allowed at a time.

    Note that objects and containers are now deleted in separate passes, to
    reduce the likelihood of 409 Conflict responses when deleting
    containers.

    Upgrade Consideration
    =====================
    If operators have enabled the bulk or slo middlewares and would like to
    preserve the prior (single-threaded) DELETE behavior, they must add the
    following line to their [filter:slo] and [filter:bulk] proxy config
    sections:

       delete_concurrency = 1

    This may be done prior to upgrading Swift.

    UpgradeImpact
    Closes-Bug: 1524454
    Change-Id: I128374d74a4cef7a479b221fd15eec785cc4694a

commit 226557afc42c245e050d84162497f46341407ef7
Author: Tim Burke <email address hidden>
Date: Thu May 19 18:55:40 2016 -0700

    Turn on H703, so our translators don't punch us

    Change-Id: I4ce3068f79563e4d4296c6e1078bc12f0cf84c96
    Related-Bug: 1559431

commit 7b706926a8ed5bbcec3a678e868e301c9a6ed8f1
Author: Alistair Coles <email address hidden>
Date: Mon May ...

This issue was fixed in the openstack/swift 2.7.0 release.