Changing encryption root secret causes container listings to 500

Hitting a similar issue (without changing the encryption root secret)

Steps to reproduce:
1. Enable encryption
2. Create container and enable versioning X-Versions-Location (pop on delete)
3. Enable object expiry (for test setting expiry to five minutes and changing object-expirer interval to 5 minutes)
4. Write object three times
5. Wait for the object to be expired
6. After that get on the container fails with error 500.

HTTP/1.1 500 Internal Error
Content-Length: 17
Content-Type: text/plain
X-Trans-Id: tx513cc4805d3e40f6a1a41-005cd1288e
Date: Tue, 07 May 2019 06:41:19 GMT

An error occurred

7. Get on the object succeeds but shows
HTTP/1.1 200 OK
Etag: ªâ¸ÚŸE`ßeä–—˜œ'üçToi
X-Object-Meta-S3B-Last-Modified: ^GZ>“ë¯Të55¢ß?

Partial Stack trace:
File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 1196, in _app_call#012 resp = self.app(env, self._start_response)#012 File "/usr/lib/python2.7/dist-packages/swift/common/middleware/crypto/decrypter.py", line 463, in __call__#012 return handler(req, start_response)#012 File "/usr/lib/python2.7/dist-packages/swift/common/middleware/crypto/decrypter.py", line 384, in handle_get#012 app_resp = handler(req, app_resp)#012 File "/usr/lib/python2.7/dist-packages/swift/common/middleware/crypto/decrypter.py", line 408, in process_json_resp#012 for obj_dict in body_json])#012 File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps#012 return _default_encoder.encode(obj)#012 File "/usr/lib/python2.7/json/encoder.py", line 207, in encode#012 chunks = self.iterencode(o, _one_shot=True)#012 File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode#012 return _iterencode(o, 0)#012UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position 2: invalid start byte (client_ip: 150.50.2.137)

Note: This doesn't happen if user deletes the object. This only happens when the object expires. Maybe something to do with the internal_client object expirer uses to delete object.

Bhaskar Singhal please share your object-expirer/internal-client pipeline. I'd like to see how they compare to the documented examples [1, 2].

1. https://github.com/openstack/swift/blob/master/etc/object-expirer.conf-sample#L92
2. https://github.com/openstack/swift/blob/master/etc/internal-client.conf-sample#L27

Swift version Rocky

Object-expirer pipeline:

[pipeline:main]
pipeline = catch_errors cache versioned_writes proxy-server

I don't have internal-client.conf

BTW, if I add keymaster/encryption in the object-expirer pipeline, then I don't see the issue, but I have not been able to find this in any documentation.

[pipeline:main]
pipeline = catch_errors cache versioned_writes keymaster encryption proxy-server

Another way to reproduce that error (with swift-2.25.0):

- forget to insert 'copy' middleware before slo/dlo/encryption in the proxy pipeline
- copy a file from some_container to new_container
- upload a file to new_container
- observe that the listing breaks because it tries to decrypt using either some_container or new_container as part of the key; and it will fail on one of the files

Deleting one of the two files makes the file listing work again.

Related log messages:

    Path stored in meta ('/.../breakage-test/problem.txt')
      does not match path from request ('/.../breakage-test-dest/problem.txt')!
    Using path from meta. (txn: <transactionid>)

Posted here to help the next guy.