adt-run doesn't realize the phone is not rebooting

With 3.13 the error is different:

~/qa/autopkgtest/3.13/run-from-checkout -d -B --unbuilt-tree . -o ~/ci/unity8/results-$(date +%Y-%m-%d-%H-%M) --- ssh -s adb -- -p 0000 --serial 04f55a0bc855b2bc

ends with:

adt-run: DBG: testbed command exited with code 0
adt-run [16:59:47]: test process requested reboot with marker 1
adt-run: DBG: sending command to testbed: reboot
invalid command reboot
Exit request sent.
<VirtSubproc>: failure: setup script reboot failed with code 1
adt-run: DBG: bomb unexpected eof from the testbed
adt-run: DBG: testbed stop
adt-run: DBG: testbed close, scratch=/tmp/adt-run.KdCUD2
adt-run: DBG: sending command to testbed: close
adt-run: DBG: bomb cannot send to testbed: ['BrokenPipeError: [Errno 32] Broken pipe\n']
adt-run: DBG: testbed stop
adt-run [16:59:51]: ERROR: testbed failed: cannot send to testbed: ['BrokenPipeError: [Errno 32] Broken pipe\n']
adt-run: DBG: testbed stop
adt-run: DBG: / tests-tree rmtree /home/vila/ci/unity8/results-2015-05-13-16-57/tests-tree

Forgot to mention that both runs were against a mako devel-proposed/195 with a unity8 update in the overlay ppa.

Re-running with image 196 on mako (with the up-to-date unity8) exhibit the same issues for both 3.14 and 3.13.

The same issue occurs with krillin devel-proposed/210.

If I adb shell to the phone (mako or krillin) and do 'sudo reboot' the phone reboots :-}

=== modified file 'lib/VirtSubproc.py'
--- lib/VirtSubproc.py 2015-05-11 08:39:02 +0000
+++ lib/VirtSubproc.py 2015-05-14 10:07:42 +0000
@@ -328,7 +328,7 @@
         adtlog.info('state saved, waiting for testbed to reboot...')
     else:
         execute_timeout(None, 30,
- auxverb + ['sh', '-c', '(sleep 3; reboot) &'])
+ auxverb + ['sh', '-c', 'reboot'])

works.

But there is probably a good reason to use this form :-/

Note that I got there by running:

phablet@ubuntu-phablet:/tmp$ SUDO_ASKPASS=/tmp/sudo_askpass.yjKC sudo -A /tmp/adt-run-wrapper sh -c (sleep 3; reboot)
t-run-wrapper sh -c (sleep 3; reboot)
bash: syntax error near unexpected token `('

Oops

phablet@ubuntu-phablet:/tmp$ SUDO_ASKPASS=/tmp/sudo_askpass.yjKC sudo -A /tmp/adt-run-wrapper sh -c '(sleep 3; reboot)'
t-run-wrapper sh -c '(sleep 3; reboot)'

that one hangs

So the patch above just got rid of the sleep and the '&' but they may be needed in other contexts.

So this should be fixed in 3.14.1 with http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=cd2478bbe53

Vincent says 3.14.1 is still broken. So when exactly did you see the regression?

> So when exactly did you see the regression?

When I tried to re-run the test in https://code.launchpad.net/~canonical-platform-qa/unity8/1421009-hangs-on-boot/ with 3.14.1

I *tried* to come back to a working version, both on autopkgtest itself and and phones, using older images, but couldn't find a working combination anymore :-/

So I went forward instead and found that s/(sleep 3; reboot) &/reboot/ worked.

Meh, this is even reproduced by my own test suite: SshRunnerNoScript.test_reboot, silly me!

This happens because of the ssh runner's terminal_kill_wrapper(), which kills all backgrounded processes that still hold ssh's stdout/stderr open. I. e. this kills the backgrounded shell which runs the reboot command.

Fixed: http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=459a28c4

Thanks for the explanation, makes sense !

Confirmed it works now.

This bug was fixed in the package autopkgtest - 3.14.2

---------------
autopkgtest (3.14.2) unstable; urgency=medium

  * Add tests/testpkg-{simple,reboot,reboot-prepare}: Very simple stub
    packages for manual testing.
  * VirtSubproc, cmd_reboot(): Disconnect the backgrounded reboot command from
    stdout/err, to avoid it getting killed by the ssh runner. (LP: #1454735)

 -- Martin Pitt <email address hidden> Tue, 19 May 2015 14:45:34 +0200

Hmm, sorry for the late reaction but, is the 'sleep 3' still needed now that the subprocess is not killed by the ssh kill wrapper ?

Vincent Ladeuil [2015-05-19 19:43 -0000]:
> Hmm, sorry for the late reaction but, is the 'sleep 3' still needed now
> that the subprocess is not killed by the ssh kill wrapper ?

Yes, it's needed. Under systemd reboot is a bit faster or behaves
differently, and causes ssh to fail. We want to orderly finish running
the reboot command remotely.

Ack, thanks for the explanation !