debootstrap fails to remove own log file on NFS

could you attach the output of an ltsp-build-client --debug run from that machine (with this proxy setup) ?

Rather vanilla log, attached.

p.s. the wiki page is part of a Kerberos LTSP setup, hence the hostname/IP configuration, and the lts.conf matches the VMware demo config so I rolled back a couple of your changes. ltsp-build-client doesn't work without a proxy too, I have a system in China with the same build problem hence the work around comments.

right, but simply skipping debootstrap will not give the expected results ...
the first step after debpootstrap is the call to apt in the chroot, since it doesnt know about the proxy it must fail.

i have plenty of reports from different parts of the world where it works fine without a proxy, did you try the proxyless setup from a CD or different mirrors ?

There is a proxy but direct connection works too, I've tried with and without proxies and both fail. The final LTSP client works correctly though. If its not clear ltsp-build-client is run twice, once including debootstrap and then the entire script exits, then a second time with debootstrap commented out which effectively continues the normal script operation.

yes, i totally understand what you are doing, but the deboostrap step is needed. you might miss some essential package. the intresting part is that it works for anybody else, i'd like to find out whats special here.

Just checking a few builds to see if its the locale setting, en_HK isn't liked too much.

Alternatively nfs locking, ...

Am Dienstag, den 10.10.2006, 13:36 +0000 schrieb StevenMcCoy:
> Just checking a few builds to see if its the locale setting, en_HK isn't
> liked too much.
>
try running:

LC_ALL=C sudo ltsp-build-client

that should ignore the locale

btw, did you notice that your debootstrap attempts the unpack and configure steps several times ? it seems there is a package missing from the mirror or something so it cant finish properly.

and... which version of ltsp-server do you have installed ?

Locale didn't make a difference.

ltsp-server 0.121

very weird, i just did a successful ltsp-build-client run (just to be sure) with exactly this package ... as well as an edubuntu test install of the 20061008 daily iso, that has the 0.118 version installed ... both worked like a charm.

aha, I just tried a loopback device and it worked correctly. Something in ltsp-build-client is not NFS friendly.

hi,
Am Dienstag, den 10.10.2006, 14:53 +0000 schrieb StevenMcCoy:
> aha, I just tried a loopback device and it worked correctly. Something
> in ltsp-build-client is not NFS friendly.
>
what do you mean with "I just tried a loopback device" ?
and no, ltsp-build-client doesnt use nfs anywhere...

My LTSP server is diskless, ltsp-build-client fails when /opt/ltsp/i386 is on nfs.

A loopback device, as in this:

$ sudo dd if=/dev/zero of=/tmp/loopfs bs=1M count=1024
$ sudo losetup /dev/loop0 /tmp/loopfs
$ sudo mkfs -t ext2 /dev/loop0
$ sudo mount /dev/loop0 /opt/ltsp
$ sudo ltsp-build-client

Am Dienstag, den 10.10.2006, 15:29 +0000 schrieb StevenMcCoy:
> My LTSP server is diskless, ltsp-build-client fails when /opt/ltsp/i386
> is on nfs.
>
> A loopback device, as in this:
>
> $ sudo dd if=/dev/zero of=/tmp/loopfs bs=1M count=1024
> $ sudo losetup /dev/loop0 /tmp/loopfs
> $ sudo mkfs -t ext2 /dev/loop0
> $ sudo mount /dev/loop0 /opt/ltsp
> $ sudo ltsp-build-client
aha, you didnt mention this in your initial report ...
but i still dont see a reason that could make it fail ... i'm building
the powerpc chroots on i386 servers by exporting /opt/ltsp readwrite and
running ltsp-build-client on a mounted nfs share from a liveCD on the
client. that works here ...

The NFS is from FreeBSD (FreeNAS), I will try with a Linux export next.

From Ubuntu exporting to Ubuntu I get the same problem, what export & mount options did you use?

exports:
/tmp *(rw,no_root_squash,async)

$ sudo mount 10.0.0.69:/tmp/ /opt/ltsp/

Oh lame, I added "set -x" to debootstrap and found the problem:

+ [ = true ]
+ rm -rf /opt/ltsp/i386/debootstrap
rm: cannot remove directory `/opt/ltsp/i386/debootstrap': Directory not empty
+ exit_function

Adding --keep-debootstrap-dir to the debootstrap line caused it to work on nfs. Now why is rm being so strange?

Great a coreutils regression. Google has a lot on this one. Version:

5.96-5ubuntu4

I experienced the same on Dapper which has this version:

5.93-5ubuntu4

That or a problem with sync on NFS, I'll generate some more detail pre-"rm -rf".

current directory: /usr/share/ltsp/plugins/ltsp-build-client/Ubuntu
ls -la:
total 98
drwxr-xr-x 2 root root 512 Oct 6 17:29 .
drwxr-xr-x 21 root root 512 Oct 6 17:38 ..
-rw-r--r-- 1 root root 83066 Oct 6 17:41 debootstrap.log
-rw-r--r-- 1 root root 10752 Oct 6 17:34 debpaths
find:
/opt/ltsp/i386/debootstrap
/opt/ltsp/i386/debootstrap/debootstrap.log
/opt/ltsp/i386/debootstrap/debpaths
rm -rf:
rm: cannot remove directory `/opt/ltsp/i386/debootstrap': Directory not empty

Deleting the debootstrap.log before the rm -rf highlights that the problem is with the NFS temporary file:

rm: cannot remove `/opt/ltsp/i386/debootstrap/.nfs00040c7200000102': Device or resource busy

So would be this categorised as a known deficiency in debootstrap/rm and hence ltstp-build-client should be updated ala:

--- 010-debootstrap.original 2006-10-06 19:03:14.000000000 +0800
+++ 010-debootstrap 2006-10-06 19:02:46.000000000 +0800
@@ -7,6 +7,7 @@
             DEBOOTSTRAPOPTS="$DEBOOTSTRAPOPTS --include=$INCLUDE"
         fi
         # Install base packages
- LC_ALL=C debootstrap $DEBOOTSTRAPOPTS --arch $ARCH $DIST $ROOT $MIRROR
+ LC_ALL=C debootstrap $DEBOOTSTRAPOPTS --keep-debootstrap-dir --arch $ARCH $DIST $ROOT $MIRROR
+ rm -rf "$ROOT/debootstrap"
         ;;
 esac

Feedback from coreutils & debootstrap teams welcome.

An alternative work around would be for debootstrap to move the log outside of the debootstrap directory, unlink, then "rm -rf" the debootstrap directory. This would leave the logs .nfs* file in the chroot root until debootstrap completes:

    if [ -e $TARGET/debootstrap/debootstrap.log ]; then
      mv $TARGET/debootstrap/debootstrap.log $TARGET/debootstrap.log
      rm $TARGET/debootstrap.log
    fi
    rm -rf "$TARGET/debootstrap"

Or just update the log copy section to move:

  if [ -e $TARGET/debootstrap/debootstrap.log ]; then
    if [ "$KEEP_DEBOOTSTRAP_DIR" = true ]; then
      cp $TARGET/debootstrap/debootstrap.log $TARGET/var/log/bootstrap.log
    else
      mv $TARGET/debootstrap/debootstrap.log $TARGET/var/log/bootstrap.log
    fi
  fi
  sync

The mv on !keep works reasonably well, attached as a patch.

This isn't a coreutils bug; it genuinely can't remove all the files (counting the NFS dotfile) and therefore can't do the rmdir(). If anywhere, the bug is that the NFS server is implementing access to still-open unlink()ed files by means of a dotfile in the same directory as the link, which means you can't do unlink() and then rmdir() without closing any fds you have open on the file as well.

This suggests a different workaround in debootstrap; I'll post a suggested patch shortly.

Actually, on reflection, the log file really needs to stay open for a log message after that point, so I like Steven's suggestion of replacing cp with mv if $TARGET/debootstrap is to be removed. I'll apply this patch.

debootstrap (0.3.3.0ubuntu7) edgy; urgency=low

  * When removing $TARGET/debootstrap, debootstrap.log is still open as
    stdout/stderr and needs to remain so, but after unlinking it some NFS
    servers implement this by a temporary file in the same directory, which
    makes it impossible to rmdir that directory. Moving it instead works
    around the problem (thanks, Steven McCoy; closes: Malone #65003).

 -- Colin Watson <email address hidden> Thu, 12 Oct 2006 09:40:44 +0100