rsync 3.2.x in Groovy depends on broken libxxhash 0.7.x

You misread my comments. It's only the 128-bit hash that depends on 0.8.0. The 0.7 version works fine with rsync, giving it the 64-bit and 32-bit hashes.

That said, of course, 0.8.0 is the better choice for the most possible features (if it is available).

Hi Wayne! Thanks for commenting.

> It's only the 128-bit hash that depends on 0.8.0.
> The 0.7 version works fine with rsync, giving it
> the 64-bit and 32-bit hashes.

Yes. Except it seems that if you switch the libxxhash0 from 0.8 to 0.7, you get different behaviour.

rsync doesn't check what kind of xxh128 is produced, so we end up with a mismatch.

Steps to reproduce:

  wget -q https://launchpad.net/ubuntu/+archive/primary/+files/rsync_3.2.3-2ubuntu1_amd64.deb

  wget -q https://launchpad.net/ubuntu/+archive/primary/+files/libxxhash0_0.7.3-1_amd64.deb

  wget -q https://launchpad.net/ubuntu/+archive/primary/+files/libxxhash0_0.8.0-1ubuntu1.20.10.1_amd64.deb

focal-node-1:

  sudo dpkg -i libxxhash0_0.7.3-1_amd64.deb \
    rsync_3.2.3-2ubuntu1_amd64.deb
  touch empty-file.txt
  echo A > non-empty-file.txt

focal-node-2:

  sudo dpkg -i rsync_3.2.3-2ubuntu1_amd64.deb \
    libxxhash0_0.8.0-1ubuntu1.20.10.1_amd64.deb
  rsync -v --debug=nstr \
    focal-node-1:*empty-file.txt \
    .

Result:

  Client negotiated checksum: xxh128
  empty-file.txt
  WARNING: empty-file.txt failed verification
    -- update discarded (will try again).
  non-empty-file.txt
  WARNING: non-empty-file.txt failed verification
    -- update discarded (will try again).
  empty-file.txt
  ERROR: empty-file.txt failed verification
    -- update discarded.
  non-empty-file.txt
  ERROR: non-empty-file.txt failed verification
    -- update discarded.

  sent 104 bytes received 255 bytes 239.33 bytes/sec
  total size is 2 speedup is 0.01
  rsync error: some files/attrs were not transferred
    (see previous errors) (code 23) at main.c(1816)
    [generator=3.2.3]

focal-node-2:

  $ ls *empty*
  ls: cannot access '*empty*': No such file or directory

I don't mind if I don't get xxh128 and get some poorer hash. But I _do_ mind if I get a hash that produces different results.

If I install libxxhash0 0.7.3 on both: things work.

If I install libxxhash0 0.8.x on both: things work.

But when there is a mismatch, things break. And uselessly too. I ended up syncing lots of GBs multiple times because our job kept retrying.

I hope that clarifies the situation.

Walter

P.S. Alternative solutions could be:
- not exporting xxh128 functions from libxxhash0 0.7.3 (but it might be a bit late for that);
- checking that xxh128 produces sane values in rsync before choosing that option.

Hello Walter,

Thanks for filling the bug and helping in making the Ubuntu server better.

However, if I get everything right, I think you're mistaken about how it works and I am sorry but what you're trying to do is not correct! You cannot just decide to take one of the package from another release and the other from another release. That is not how it works, I am afraid :/

If you do a fresh installation of rsync on Groovy, everything works fine! See here:

$ lxc launch images:ubuntu/groovy groovy-rsync
$ lxc shell groovy-rsync
# apt update && apt install rsync
  ### here you'll see that the right version of libxxhash gets installed.
      (cf: `Setting up libxxhash0:amd64 (0.8.0-1ubuntu1.20.10.1) ...)
# ls -lah /usr/lib/x86_64-linux-gnu/libxxhash.so.0
lrwxrwxrwx 1 root root 18 Jan 12 11:17 /usr/lib/x86_64-linux-gnu/libxxhash.so.0 -> libxxhash.so.0.8.0

...which is linked against the right version. Further libxxhash-dev/0.8.0-1 is not in Focal but in Groovy, so the patch will not be right to apply in Focal. That said, I agree that it'd have been nice to have that version constraints but I don't see this as something that'd cause problems unless somebody tries to do some sort of manual intervention. :)

Given this, I am inclined towards believing that this is not really a bug in rsync but a "local" issue (that is, manually installing a version of rsync from Groovy to Focal and expecting it to work w/ libxxhash-dev). So I am setting this to "Incomplete" for now, leaving some space for discussion - in case I wrongly interpreted your problem and thus this report.

But should you feel that it's still a bug, please set the status back to "New" along with some reasoning (and hopefully a reproducer!) as to why you think that it's indeed a bug rather than a local issue. Thank you, again! \o/

How about this case:

- I make a hypothetical package that depends on libxxhash < 0.8 because I want the "broken/old" xxh128 support;

- I have libxxhash 0.7.3 (that came with Focal);

- I have rsync 3.1.x (that came with Focal);

- Now I release-upgrade my system from Focal to Groovy;

- I get all kinds of new packages, including rsync 3.2.3;

- Dependency resolution fixes that I _don't_ get libxxhash 0.8 _because_ I have:

- (a) my hypothetical package pinned and it depends on libxxhash<0.8;

- (b) rsync that depends on libxxhash>=0.7.1.

- The dependencies are all still satisfied and I get to keep libxxhash=0.7.3.

In this new situation I have completed the upgrade to Groovy. As long as my hypothetical package that depends on libxxhash <0.8 exists, Ubuntu will not upgrade libxxhash.

And then I'm in the same situation as I am now:

root@groovy-rsync:~# apt-mark hold hypothetical-pkg
hypothetical-pkg set on hold.

root@groovy-rsync:~# apt-cache show hypothetical-pkg | grep ^Depe
Depends: libxxhash0 (<= 0.8)

root@groovy-rsync:~# apt-get install rsync
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  openssh-server
The following NEW packages will be installed:
  rsync

root@groovy-rsync:~# dpkg -l | grep -E 'rsync|libxxhash'
hi hypothetical-pkg 1.0
ii libxxhash0:amd64 0.7.3-1
ii rsync 3.2.3-2ubuntu1

root@groovy-rsync:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  libxxhash0
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

This may not be the commonest of cases. But this would be an impossible situation if rsync_3.2.3 simply depended on 0.8+ instead of on 0.7.1+. (An impossible situation is good in this case, because you have to choose between either hypothetical-pkg or rsync.)

So, I still feel that this is a bug in the rsync control file in Groovy. Not a bug in the rsync source.

(Ok, in fact, I think it's ultimately a bug in soname-version/symbol handling of libxxhash. But that's not where the problem manifests itself.)

I'll leave it as is if you still feel it should be closed. But at least it has some visibility/presence on the internet so others are helped if they also run into this issue.

Cheers :)
Walter

Sorry, the use cases you describe are not supported by Ubuntu.

You're welcome to hack your system as you wish, but that doesn't mean that we will necessarily make changes in Ubuntu to accommodate that. We do try to be helpful, of course. And in this case I agree that it is a bug that rsync doesn't depend on a higher version of libxxhash automatically. That's something we should fix, but it is of low priority but I don't expect such a fix to be backported to Groovy because, as presented at the moment, this doesn't meet our threshold of disrupting users to achieve such a change. See https://wiki.ubuntu.com/StableReleaseUpdates for details of our policies in this area.

In particular:

> - I make a hypothetical package that depends on libxxhash < 0.8 because I want the "broken/old" xxh128 support

Such a hypothetical package does not exist in the Ubuntu archive in practice. Adding third party packages is not something that Ubuntu can realistically support. The scenario you present is exactly why we don't support third party packages in the general case: they break distribution release upgrades.

> - libxxhash-dev,
> + libxxhash-dev (>= 0.8),

I don't think your patch will work as-is. You've changed the build dependency versioning, not the binary package dependency versioning. I suspect what needs adjusting is the symbols file in the xxhash source (debian/libxxhash0.symbols). However this needs further investigation and the low priority of this issue means that I'm not going to spend any more time on this unless a more important and supported use case is presented here.

> (Ok, in fact, I think it's ultimately a bug in soname-version/symbol handling of libxxhash. But that's not where the problem manifests itself.)

Right. We track bugs and their fixes by following the status of a fix against the root cause. So I'm going to reassign the bug to that package as it seems likely to me that this is where the problem lies.

> I'll leave it as is if you still feel it should be closed.

I agree that it's not correct that the binary dependency is wrong, so this bug can remain open if somebody wants to volunteer a fix. However, I suggest you find the fix and send it to wherever the problem originates in our ecosystem (maybe Debian? Or perhaps xxhash upstream?). Unless a supported use case is presented, I think it's unlikely that we'll carry a patch for this in Ubuntu.

> But at least it has some visibility/presence on the internet so others are helped if they also run into this issue.

Sure. People affected by this are welcome to coordinate in this bug.

I'm going to explicitly mark a Groovy task as Won't Fix to make it clear that we don't expect any change will be made in Groovy to fix this. The bug remains open to being fixed in a future Ubuntu release if a volunteer takes the appropriate steps to get the issue resolved at its actual origin.

> So I'm going to reassign the bug to that package as it seems likely to me that this is where the problem lies.

Actually I'm not sure it's a symbol versioning issue now. Looking at your error, perhaps it's a protocol negotiation issue in rsync? Either way, it needs further investigation by a volunteer, and the rest of my previous comment stands - I'd appreciate a fix at the origin of the problem, and that would go into a subsequent Ubuntu release - but I don't think it's appropriate to fix in Groovy based on the use case presented.

Anyone who has an rsync that supports the xxh128 hash using the xxhash 0.7 library did something very bad in their build. An rsync compiled with that library only supports xxh64 & xxh32. Thus, this described mismatch never occurs in a stock rsync (the 128-bit hash was unstable in xxhash 0.7, and thus rsync doesn't use it).

To be clear, if you have xxhash 0.7 and "rsync -V" reports xxh128 in the checksum list, fix the rsync build by getting rid of any patch that enabled that unsupported hash.

Thanks, Robie, for the excellent response.

> Actually I'm not sure it's a symbol versioning issue now.

Maybe this example can persuade you that there is a problem between (lib)xxhash in Focal and in Groovy:

  focal$ xxh128sum <(echo -n)
  07fd4e968e916ae11f17545bce1061f1 /dev/fd/63

  groovy$ xxh128sum <(echo -n)
  99aa06d3014798d86001c324468d497f /dev/fd/63

> I suggest you find the fix and send it to wherever
> the problem originates in our ecosystem (maybe Debian?
> Or perhaps xxhash upstream?). Unless a supported
> use case is presented, I think it's unlikely that
> we'll carry a patch for this in Ubuntu.

I think I can safely say now that the problem originates in the Debian package:

if the symbols file is altered as I have done here [1] the problem goes away:
[1] https://github.com/norbusan/debian-xxhash/issues/3
```
--- a/debian/libxxhash0.symbols
+++ b/debian/libxxhash0.symbols
@@ -1,9 +1,9 @@
 libxxhash.so.0 libxxhash0 #MINVER#
- XXH128@Base 0.7.0
- XXH128_canonicalFromHash@Base 0.7.1
+ XXH128@Base 0.8.0
+ XXH128_canonicalFromHash@Base 0.8.0
...
```
A rebuild of rsync against a libxxhash0 with the above symbol changes will properly Depend on 0.8.0+.

I am just not intimately familiar with the rules for these symbol files, so I'm not 100% sure if the above changes are "legal". Can you shed some light on this?

> I'd appreciate a fix at the origin of the problem,
> and that would go into a subsequent Ubuntu release

Well. If we get the symbols file updated so it says 0.8.0, then subsequent updates from Debian would trickle into Ubuntu. Then any _new_ builds (of for example rsync) against the updated libxxhash library package would get the dependencies fixed.

That does leave the problem of libxxhash 0.7.x on Ubuntu/Focal:

- it has a broken xxh128sum binary
- it exposes access to old/invalid xxh128 functions leading to possible future problems when people start upgrading their libxxhash0

To remedy that, I would suggest a rebuild of xxhash + libxxhash0 on Focal where all xxh128 symbols/functionality is removed/blacklisted.

I could do some quilting that simply `#if 0`'s the relevant code. Is this something that would be accepted?

Thank you for the further investigation.

It sounds like upstream broke ABI between libxxhash0 0.7 and libxxhash0 0.8 but did not bump the soname when they did so. Perhaps that's because upstream still considers this experimental as it has a version < 1.0, or perhaps that was a mistake. For a proper fix in the development release, I'd expect upstream to bump the soname to make the symbol's behaviour unambiguous. Without that, proper ABI compatibility is impossible since different distributions and releases will have different behaviour for the same soname and symbol depending on the version used to build the library.

> To remedy that, I would suggest a rebuild of xxhash + libxxhash0 on Focal where all xxh128 symbols/functionality is removed/blacklisted.

I'm not convinced. The use case you've presented is still of low/unsupported priority and not properly resolved upstream. On the other hand, we try not to break ABI in a stable release. And as pointed out:

> Results produced by v0.7.x are not comparable with results
> from v0.7.y. However, the API is completely stable, and it
> can safely be used for ephemeral data (local sessions).

...so if a user has built against libxxhash for such a use case that is supported by upstream, your proposal for changing Focal would break them.

Leaving Focal as-is still seems like the least-worst answer to me.

> ...so if a user has built against libxxhash for such
> a use case that is supported by upstream, your proposal
> for changing Focal would break them.

*shrug*

I'd rather have my build break now than succeed and get harder to debug issues with failing cross device communication.

Well, I've done my part. Let me know if there's something I can do. I'll consider this closed on my end :)

This bug was fixed in the package xxhash - 0.8.1-1

---------------
xxhash (0.8.1-1) unstable; urgency=medium

  * New maintainer. Closes: #1003524.
  * debian/copyright: Update copyright holder years for debian/*.
  * debian/control: Document Rules Requires Root as no.
  * Update package to use debhelper-compat = 13.
  * Bump Standards-Version to 4.6.0. No changes required.

 -- Josue Ortega <email address hidden> Fri, 14 Jan 2022 08:49:06 -0600