Mixed versions of liberasurecode leads to quarantined fragments

Note that this is only a problem *if your servers were using libec's alternative CRC*. If they're using zlib pre-upgrade, the upgrade should be seamless.

Also note that changing the order in which you upgrade things isn't sufficient to avoid the issue:

 * If you upgrade all object servers before all proxies, you may still have the reconstructor write fragments that the proxies can't read.
 * When upgrading a fleet of proxies, you will have upgraded proxies writing down data that not-yet-upgraded proxies can't read.

Of course, when the proxy can't read the frag, it won't quarantine anything (for obvious reasons). It'll respond 500 and log tracebacks like

Jul 2 19:11:27 saio proxy-server: Error decoding fragments for '/AUTH_test/cont/obj':
Traceback (most recent call last):
  File ".../swift/proxy/controllers/obj.py", line 1430, in _decode_segments_from_fragments
    segment = self.policy.pyeclib_driver.decode(fragments)
  File ".../pyeclib/ec_iface.py", line 292, in decode
    force_metadata_checks)
  File ".../pyeclib/core.py", line 87, in decode
    fragment_len, ranges, force_metadata_checks)
pyeclib.ec_iface.ECInvalidFragmentMetadata: pyeclib_c_decode ERROR: Fragment integrity check failed. Please inspect syslog for liberasurecode error report. (txn: tx2dd3ad4ff37047689a3b8-005efe315f) (client_ip: 127.0.0.1)
Jul 2 19:11:27 saio proxy-server: ERROR Unhandled exception in request:
Traceback (most recent call last):
  File ".../swift/proxy/server.py", line 572, in handle_request
    return handler(req)
  File ".../swift/proxy/controllers/base.py", line 291, in wrapped
    return func(*a, **kw)
  File ".../swift/proxy/controllers/obj.py", line 264, in GET
    return self.GETorHEAD(req)
  File ".../swift/proxy/controllers/obj.py", line 252, in GETorHEAD
    resp = self._get_or_head_response(req, node_iter, partition, policy)
  File ".../swift/proxy/controllers/obj.py", line 2426, in _get_or_head_response
    app_iter.kickoff(req, resp)
  File ".../swift/proxy/controllers/obj.py", line 1085, in kickoff
    self.stashed_iter = reiterate(self._real_iter(req, resp.headers))
  File ".../swift/common/utils.py", line 4010, in reiterate
    chunk = next(iterator)
  File ".../swift/common/utils.py", line 4711, in string_along
    for x in useful_iter:
  File ".../swift/proxy/controllers/obj.py", line 1366, in _iter_one_range
    for i, next_seg in enumerate(segment_iter):
  File ".../swift/proxy/controllers/obj.py", line 1430, in _decode_segments_from_fragments
    segment = self.policy.pyeclib_driver.decode(fragments)
  File ".../pyeclib/ec_iface.py", line 292, in decode
    force_metadata_checks)
  File ".../pyeclib/core.py", line 87, in decode
    fragment_len, ranges, force_metadata_checks)
pyeclib.ec_iface.ECInvalidFragmentMetadata: pyeclib_c_decode ERROR: Fragment integrity check failed. Please inspect syslog for liberasurecode error report. (txn: tx2dd3ad4ff37047689a3b8-005efe315f) (client_ip: 127.0.0.1)

Wrote up a utility to check which CRC (if any) was used for provided fragments. Generally, if you're running liberasurecode 1.2.0 or later (which introduced metadata checksumming four years ago), fragment CRCs will either be "legacy" or "zlib". Sample several from different servers to establish a consensus. On older clusters, you may also encounter fragments where the CRC is "not set"; ignore these.

If all nodes are currently running liberasurecode>1.5.0, you are *not affected*; all nodes can already read fragments with either CRC. If nodes are running liberasurecode<=1.5.0 and

  * fragments have "zlib" CRCs, you are *not affected* and can upgrade safely.
  * fragments have "legacy" CRCs, you *are* affected and will need to upgrade carefully.

If you are running liberasurecode<=1.5.0 and there's a mix of "zlib" and "legacy" CRCs, more investigation is necessary.

Reviewed: https://review.opendev.org/738959
Committed: https://git.openstack.org/cgit/openstack/liberasurecode/commit/?id=5626cd5791bd35279594e067c10581e89ae66abc
Submitter: Zuul
Branch: master

commit 5626cd5791bd35279594e067c10581e89ae66abc
Author: Tim Burke <email address hidden>
Date: Wed Jul 1 21:59:45 2020 -0700

    Be willing to write fragments with legacy crc

    ...if users *really* want to. They opt-in at run time by setting

        LIBERASURECODE_WRITE_LEGACY_CRC=1

    in the environment; leaving it unset, set to an empty string, or set to
    the string "0" continues to write zlib crcs.

    UpgradeImpact
    =============
    This option is intended to allow a smooth upgrade from liberasurecode
    1.5.0 and earlier in a system with multiple readers and writers:

      * Before upgrade, ensure the environment variable is set on all nodes.
        This will be ignored by earlier versions.
      * Upgrade liberasurecode on each node in the system, restarting any
        services that use it. Every node continues writing CRCs that are
        still usable by nodes that have not yet upgraded.
      * Now that every node is capable of reading zlib CRCs, remove the
        environment variable from each node to start writing new CRCs.

    If you are already using 1.6.0 or later, just upgrade normally.

    Closes-Bug: #1886088
    Closes-Bug: #1867937
    Related-Bug: #1666320
    Needed-By: https://review.opendev.org/#/c/739164/
    Change-Id: I9adfbe631a2dddc592fd08f8a325f3e8331b92f1

Reviewed: https://review.opendev.org/739164
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=599f63e762f1f2b51b64353a31342be5e6371fc5
Submitter: Zuul
Branch: master

commit 599f63e762f1f2b51b64353a31342be5e6371fc5
Author: Tim Burke <email address hidden>
Date: Thu Jul 2 16:29:59 2020 -0700

    ec: Add an option to write fragments with legacy crc

    When upgrading from liberasurecode<=1.5.0, you may want to continue
    writing legacy CRCs until all nodes are upgraded and capabale of reading
    fragments with zlib CRCs.

    Starting in liberasurecode>=1.6.2, we can use the environment variable
    LIBERASURECODE_WRITE_LEGACY_CRC to control whether we write zlib or
    legacy CRCs, but for many operators it's easier to manage swift configs
    than environment variables. Add a new option, write_legacy_ec_crc, to the
    proxy-server app and object-reconstructor; if set to true, ensure legacy
    frags are written.

    Note that more daemons instantiate proxy-server apps than just the
    proxy-server. The complete set of impacted daemons should be:

      * proxy-server
      * object-reconstructor
      * container-reconciler
      * any users of internal-client.conf

    UpgradeImpact
    =============
    To ensure a smooth liberasurecode upgrade:

     1. Determine whether your cluster writes legacy or zlib CRCs. Depending
        on the order in which shared libraries are loaded, your servers may
        already be reading and writing zlib CRCs, even with old
        liberasurecode. In that case, no special action is required and
        WRITING LEGACY CRCS DURING THE UPGRADE WILL CAUSE AN OUTAGE.
        Just upgrade liberasurecode normally. See the closed bug for more
        information and a script to determine which CRC is used.
     2. On all nodes, ensure Swift is upgraded to a version that includes
        write_legacy_ec_crc support and write_legacy_ec_crc is enabled on
        all daemons.
     3. On each node, upgrade liberasurecode and restart Swift services.
        Because of (2), they will continue writing legacy CRCs which will
        still be readable by nodes that have not yet upgraded.
     4. Once all nodes are upgraded, remove the write_legacy_ec_crc option
        from all configs across all nodes. After restarting daemons, they
        will write zlib CRCs which will also be readable by all nodes.

    Change-Id: Iff71069f808623453c0ff36b798559015e604c7d
    Related-Bug: #1666320
    Closes-Bug: #1886088
    Depends-On: https://review.opendev.org/#/c/738959/

This issue was fixed in the openstack/swift 2.27.0 release.