"Invalid cross-device link" and failed to preserve ownership error on image-output...

Hi David,

Thanks for the bug report. So, 'cp -p' should only fail if the target filesystem is not a POSIX filesystem. Is that the case for you? I see /home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img as a destination, I'm not sure if 'hgfs' refers to a particular filesystem with non-POSIX semantics?

This is certainly a legitimate bug. We may be able to drop the '-p' with the reasoning that a file copy will create the new file with the same umask and owner as the original file creation, but we need to check this assumption.

Hgfs is a file system that VMware fusion uses to mount a host file system
from OS X onto the guest...

On Friday, October 28, 2016, Steve Langasek <email address hidden>
wrote:

> Hi David,
>
> Thanks for the bug report. So, 'cp -p' should only fail if the target
> filesystem is not a POSIX filesystem. Is that the case for you? I see
> /home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/
> Signage/flock_installer
> /ics-ubuntu-image.img as a destination, I'm not sure if 'hgfs' refers to
> a particular filesystem with non-POSIX semantics?
>
> This is certainly a legitimate bug. We may be able to drop the '-p'
> with the reasoning that a file copy will create the new file with the
> same umask and owner as the original file creation, but we need to check
> this assumption.
>
> ** Changed in: ubuntu-image
> Status: New => Triaged
>
> ** Changed in: ubuntu-image
> Importance: Undecided => Medium
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1637554
>
> Title:
> "Invalid cross-device link" and failed to preserve ownership error on
> image-output...
>
> Status in Ubuntu Image:
> Triaged
>
> Bug description:
> Here is your wall of traceback:
>
> cro@ubuntu:~$ sudo /snap/bin/ubuntu-image \
> > --extra-snaps=/home/cro/home_root/mnt/2/snapcraft_kernel_
> build_test/linuxium-kernel_4.4.0-45.66_amd64.snap \
> > -c beta -o ${DIR2_}/ics-ubuntu-image.img \
> > ${DIR4_}/ics.model
> Fetching core
> Copying "/home/cro/home_root/mnt/2/snapcraft_kernel_build_test/
> linuxium-kernel_4.4.0-45.66_amd64.snap" (linuxium-kernel)
> Fetching pc
> linuxium-kernel already prepared, skipping
> COMMAND FAILED: ['cp', '-p', '--sparse=always',
> '/tmp/tmpsjo_rg8t/.images/disk.img', '/home/cro/home_root/mnt/hgfs/
> cro/Documents/TanagerProjects/Signage/flock_installer/ics-
> ubuntu-image.img']
> cp: failed to preserve ownership for '/home/cro/home_root/mnt/hgfs/
> cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img':
> Operation not permitted
> uncaught exception in state machine step: [11] finish
> Traceback (most recent call last):
> File "/snap/ubuntu-image/25/usr/lib/python3.5/shutil.py", line 538,
> in move
> os.rename(src, real_dst)
> OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpsjo_rg8t/.images/disk.img'
> -> '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/
> Signage/flock_installer/ics-ubuntu-image.img'
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/state.py",
> line 68, in __next__
> step()
> File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/builder.py",
> line 430, in finish
> shutil.move(self.disk_img, self.output, copy_function=sparse_copy)
> File "/snap/ubuntu-image/25/usr/lib/python3.5/shutil.py", line 552,
> in move
> copy_function(src, real_dst)
> File "/snap/ubuntu-image/25/lib/python3.5/site-packages/ubuntu_image/helpers.py",
> line 142, in sparse_copy
> ...

I wasn't able to get file sharing working with a Zesty guest in Fusion 8.5.3. Fusion kept complaining that it couldn't mount the host directory, so I suspect it's an incompatibility with open-vm-tools.

I *was* able to mount an afp: file system onto my Yakkety machine though and tries an equivalent command which moved the final disk image to the afp mounted directory. This reproduced the crash.

In this particular case, it looks like it's the copying of the timestamps that isn't supported. `cp -p` is equivalent to `cp --preserve=mode,ownership,timestamps` and if you substitute the former for the latter w/o timestamps (i.e. cp --preserve=mode,ownership) it works just fine. Preserving timestamps is effectively unnecessary since it's not terribly useful information and the cp is going to happen very soon after the original disk image creation anyway[*].

So I propose that simple change, although I won't be able to write a good test case for it.

[*] Unless you ran the state machine to the 'finish' step and then resumed it, but that's not a typical usage and again, I don't think preserving the timestamps is really all that useful.

@croepha: Please test to see if --preserve-mode,ownership works for you.

https://github.com/CanonicalLtd/ubuntu-image/pull/103

Actually, hmm. The cp error message is different in the afp case, so maybe hgfs has different semantics than afp after all.

Note that even with the operation not supported error, the resulting file is checksum equivalent, which means that `cp --preserve` is failing after the bits are copied. How much does it matter that the mode, ownership, and timestamp are preserved? I've already laid out why I think the timestamp doesn't matter as much. In my own tests over afp, the ownership of the new file is equivalent to the remote user's uid anyway. It's true that the remote file (after the exception) does not honor umask of either the source umask or target umask.

So another option would be: catch the CalledProcessError and if the checksums of the copied file are not equivalent to the source file, re-raise the exception. If they're equivalent, swallow the error.

Or we could just remove -p and be done with it.

Barry:

I think it has specifically to do with root, and hgfs not allowing you to
set root ownership on hgfs. examine this:

cro@ubuntu:~$ touch test2
cro@ubuntu:~$ cp -p test2 /mnt/hgfs/cro/Documents/
cro@ubuntu:~$ sudo touch test3
cro@ubuntu:~$ sudo cp -p test3 /mnt/hgfs/cro/Documents/
cp: failed to preserve ownership for '/mnt/hgfs/cro/Documents/test3':
Operation not permitted
cro@ubuntu:~$ sudo touch test4
cro@ubuntu:~$ sudo cp --preserve=mode,ownership test4
/mnt/hgfs/cro/Documents/
cp: failed to preserve ownership for '/mnt/hgfs/cro/Documents/test4':
Operation not permitted
cro@ubuntu:~$

ask you can see, -p works fine for a normal user...

I would recommend just dropping it. I'm not sure what the added benefit of
'-p' is in a practical sense.... If the permissions in question were a
priority for the the user, than that user should select umasks and
filesystems appropriately.. I think the default functionality of `cp'
simply does the right thing...

On Tue, Dec 13, 2016 at 3:30 PM Barry Warsaw <email address hidden>
wrote:

> Actually, hmm. The cp error message is different in the afp case, so
> maybe hgfs has different semantics than afp after all.
>
> Note that even with the operation not supported error, the resulting
> file is checksum equivalent, which means that `cp --preserve` is failing
> after the bits are copied. How much does it matter that the mode,
> ownership, and timestamp are preserved? I've already laid out why I
> think the timestamp doesn't matter as much. In my own tests over afp,
> the ownership of the new file is equivalent to the remote user's uid
> anyway. It's true that the remote file (after the exception) does not
> honor umask of either the source umask or target umask.
>
> So another option would be: catch the CalledProcessError and if the
> checksums of the copied file are not equivalent to the source file, re-
> raise the exception. If they're equivalent, swallow the error.
>
> Or we could just remove -p and be done with it.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1637554
>
> Title:
> "Invalid cross-device link" and failed to preserve ownership error on
> image-output...
>
> Status in Ubuntu Image:
> In Progress
>
> Bug description:
> Here is your wall of traceback:
>
> cro@ubuntu:~$ sudo /snap/bin/ubuntu-image \
> >
> --extra-snaps=/home/cro/home_root/mnt/2/snapcraft_kernel_build_test/linuxium-kernel_4.4.0-45.66_amd64.snap
> \
> > -c beta -o ${DIR2_}/ics-ubuntu-image.img \
> > ${DIR4_}/ics.model
> Fetching core
> Copying
> "/home/cro/home_root/mnt/2/snapcraft_kernel_build_test/linuxium-kernel_4.4.0-45.66_amd64.snap"
> (linuxium-kernel)
> Fetching pc
> linuxium-kernel already prepared, skipping
> COMMAND FAILED: ['cp', '-p', '--sparse=always',
> '/tmp/tmpsjo_rg8t/.images/disk.img',
> '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img']
> cp: failed to preserve ownership for
> '/home/cro/home_root/mnt/hgfs/cro/Documents/TanagerProjects/Signage/flock_installer/ics-ubuntu-image.img':
> Operation not permitted
> unc...

David, thanks for the additional information.

On Dec 14, 2016, at 01:06 AM, david butler wrote:

>I would recommend just dropping it. I'm not sure what the added benefit of
>'-p' is in a practical sense.... If the permissions in question were a
>priority for the the user, than that user should select umasks and
>filesystems appropriately.. I think the default functionality of `cp'
>simply does the right thing...

We pretty much came to the same conclusion. :) We're dropping it.