I'm solving a misplaced partition, but I get 500 error code multiple times.
Misplaced on node : d655
$ sudo /opt/ss/bin/python3 /opt/ss/support/bin/classify_handoff_parts.py all
device: primary handoff( suffix) misplaced( suffix)
.....
d655: 0 0( 0) 1( 4089)
Get 500 error from remote node
$ /opt/ss/bin/python3.7 /opt/ss/bin/swift-object-reconstructor /etc/swift/object-server/a.conf -d d655 -v -o
object-reconstructor: Starting 176727
object-reconstructor: [worker 1/1 pid=176738] Running object reconstructor in script mode.
object-reconstructor: [worker 1/1 pid=176738] X.X.X.X:6003/d38782/229849 Unexpected response: ":ERROR: 500 'ERROR: With :UPDATES: 1 failures to 0 successes'"
object-reconstructor: [worker 1/1 pid=176738] 1/3 (33.33%) partitions reconstructed in 10.93s (0.09/sec, 21s remaining)
object-reconstructor: [worker 1/1 pid=176738] Object reconstruction complete (once). (0.18 minutes)
object-reconstructor: Finished 176727
object-reconstructor: Exited 176727
Error logs on remote node
2023-02-16T03:49:55.322872+00:00 object-server-r: warning ssync subrequest failed with 409: POST /d38782/229849/XXX/XXXX/XXX (<html><h1>Conflict</h1><p>There was a conflict when trying to complete your request.</p></html>)
2023-02-16T03:55:21.584880+00:00 object-server-r: warning ssync subrequest failed with 409: POST /d38782/229849/XXX/XXXX/XXX (<html><h1>Conflict</h1><p>There was a conflict when trying to complete your request.</p></html>)
....
Object info on remote node
$ ls
XXX.data XXX.meta
$ swift-object-info XXX
Path: XXX
Account: XXX
Container: XXX
Object: XXX
Object hash: XXX
Content-Type: binary/octet-stream
Timestamp: 2023-02-10T08:24:22.774370 (1676017462.77437_0000000000000001)
System Metadata:
X-Object-Sysmeta-Allow-Reserved-Names: true
X-Object-Sysmeta-Container-Update-Override-Etag: XXX
X-Object-Sysmeta-Ec-Content-Length: 0
X-Object-Sysmeta-Ec-Etag: XXX
X-Object-Sysmeta-Ec-Frag-Index: 2
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_cauchy 8+4
X-Object-Sysmeta-Ec-Segment-Size: 1048576
X-Object-Sysmeta-Symlink-Target: XXX
X-Object-Sysmeta-Symlink-Target-Bytes: 7
X-Object-Sysmeta-Symlink-Target-Etag: XXX
X-Object-Sysmeta-Symloop-Extend: true
X-Object-Sysmeta-Versions-Symlink: true
Transient System Metadata:
No metadata found
User Metadata:
No metadata found
Other Metadata:
No metadata found
ETag: XXX (valid)
Content-Length: 0 (valid)
Partition 229849
Hash XXX
Object info on source node
$ ls
XXX.data XXX.meta
$ swift-object-info XXX
Path: XXX
Account: XXX
Container: XXX
Object: XXX
Object hash: e0766dfd7443638268add951f36d3e8b
Content-Type: binary/octet-stream
Timestamp: 2023-02-10T08:24:22.774370 (1676017462.77437_0000000000000001)
System Metadata:
X-Object-Sysmeta-Allow-Reserved-Names: true
X-Object-Sysmeta-Container-Update-Override-Etag: XXX
X-Object-Sysmeta-Ec-Content-Length: 0
X-Object-Sysmeta-Ec-Etag: XXX
X-Object-Sysmeta-Ec-Frag-Index: 2
X-Object-Sysmeta-Ec-Scheme: isa_l_rs_cauchy 8+4
X-Object-Sysmeta-Ec-Segment-Size: 1048576
X-Object-Sysmeta-Symlink-Target: XXX
X-Object-Sysmeta-Symlink-Target-Bytes: 7
X-Object-Sysmeta-Symlink-Target-Etag: XXX
X-Object-Sysmeta-Symloop-Extend: true
X-Object-Sysmeta-Versions-Symlink: true
Transient System Metadata:
No metadata found
User Metadata:
No metadata found
Other Metadata:
No metadata found
ETag: XXX (valid)
Content-Length: 0 (valid)
Partition 229849
Hash XXX
The reconstructor/ssync appears to be repeatedly trying to sync an object that is already in sync. That will cause a 409 conflict in the ssync subrequest stream.
We inspected the on disk files for this object on the sender and receiver side. Both had identical timestamps. However, the data files have an offset timestamps, and there is also a meta file.
i.e. both sender and receiver have:
t0_1#2#d.data
t1.meta
The sender encodes these timestamps in a compact form and the receiver decodes them and compares to its on disk file set.
The encoding represents t1 (the meta timestamp) as a delta from the data timestamp, NOT INCLUDING the data timestamp offset, i.e. delta = t1 - t0.
The decoding is erroneously calculating the meta timestamp as the sum of the delta plus the data timestamp INCLUDING the offset.
So the ssync receiver therefore erroneously thinks that the sender has a newer meta file:
t0_1#2#d.data
t1_1.meta
and so the receiver request that the sender POSTs its meta file content. On receiving the POST, the ssync receiver object server compares the correct timestamps and discovers they are identical to what it has on disk and so returns a 409 Conflict response.