Old clients cannot talk to Docker in 16.04

Oh fun. At the moment we don't even have Go 1.5 in Xenial and I don't really want to add it, although thanks to some re-working in 16.04 we can at least add it cleanly now.

It was pointed out to me that moving to 1.5 would mean not building for s390x, which even more isn't something we want to do. I'd be happier vendoring a hacked version of net/http into the docker source package, or whatever is required along those lines tbh.

Vendoring net/http turns out to be really annoying because you end up having to vendor lots of other stdlib packages as well to make things work. I have another, even more evil idea: binary patching! I *think* this program will allow a docker binary to accept the bogus headers sent by docker < 1.10: http://paste.ubuntu.com/16060673/

Applying https://github.com/docker/docker/pull/22000 is another option
worth considering. The patch isn't ideal (and upstream currently has
no interest in applying it), but it does do the job.

Attaching a patch (to be applied to http://anonscm.debian.org/cgit/docker/docker.io.git/log/?h=ubuntu) that fixes this in my testing.

This bug was fixed in the package docker.io - 1.10.3-0ubuntu7

---------------
docker.io (1.10.3-0ubuntu7) yakkety; urgency=medium

  [ Tianon Gravi ]
  * Update GOPATH logic in debian/tests/integration to match debian/rules

  [ Michael Hudson-Doyle ]
  * Add d/patches/22000--ignore-invalid-host-header.patch. (LP: #1574904)

 -- Tianon Gravi <email address hidden> Tue, 26 Apr 2016 16:40:15 -0700

When will this fix be released for Xenial (16.04)? I can only find 1.10.3-0ubuntu6

Thank you for your help,

Jan

I have the same question as the last comment; please could this be released for 16.04.

We hope to have a fix for this in Xenial within a week.

At mwhudson's request I'm testing this on a xenial server using the packages in ppa:mwhudson/devirt.

* set up a clean 16.04 machine and reproduced the failures I saw with weave
* added ppa:mwhudson/devirt & changed 'xenial' to 'yakkety'
* apt-get upgrade, then:

$ sudo apt-get install docker.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  containerd runc
Suggested packages:
  aufs-tools debootstrap lxc rinse zfs-fuse | zfsutils
The following NEW packages will be installed:
  containerd runc
The following packages will be upgraded:
  docker.io

Now, I get::

$ docker version
Client:
 Version: 1.11.2
 API version: 1.23
 Go version: go1.6.2
 Git commit: b9f10c9
 Built: Thu, 09 Jun 2016 20:37:27 +1200
 OS/Arch: linux/amd64
Error response from daemon: client is newer than server (client API version: 1.23, server API version: 1.22)

Manually restarting docker (sudo systemctl restart docker) seems to work around the problem.

The installation process also seemed to override my local changes to /etc/systemd/system/multi-user.target.wants/docker.service

However, after these things, when using the new docker.io package, I could no longer observe the problems I saw with weave. Success!

Hey, thanks for testing. I've reported the two issues you mention as separate bugs https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1591021 and https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1591023, happy to continue conversations there. I'll be asked you to do verification of the packages when they get to xenial-proposed I expect!

How's it going getting these into xenial-proposed?

The packages are waiting for approval by the SRU team.

Hello <email address hidden>, or anyone else affected,

Accepted docker.io into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io/1.11.2-0ubuntu4~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello <email address hidden>, or anyone else affected,

Accepted runc into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/0.1.1-0ubuntu5~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I've done the rebuild tests needed in my ppa https://launchpad.net/~mwhudson/+archive/ubuntu/devirt/+packages

With the -proposed version, I'm seeing failures on arm64, apparently due to the new dependency on containerd.

Starting Docker Socket for the API.
Listening on Docker Socket for the API.
Starting Docker Application Container Engine...
time="2016-07-01T15:45:38.219004000Z" level=info msg="New containerd process, pid: 19074\n"
time="2016-07-01T15:45:38Z" level=fatal msg="containerd: epoll wait" error="function not implemented"
[76387.156087] containerd[19079]: syscall 1069
[76387.156096] Code: f94023e4 f94027e5 f9400fe8 d4000001 (b13ffc1f)
[76387.156102] CPU: 34 PID: 19079 Comm: containerd Tainted: P O 4.4.0-29-generic #48
[76387.156105] Hardware name: Default string Default string/Default string, BIOS 5.11 12/12/2012
[76387.156111] task: ffff801f48b0a700 ti: ffff8019ed864000 task.ti: ffff8019ed864000
[76387.156117] PC is at 0x4ed338
[76387.156120] LR is at 0x4ed318
[76387.156123] pc : [<00000000004ed338>] lr : [<00000000004ed318>] pstate: 60000000
[76387.156126] sp : 00000048200687c0
[76387.156130] x29: 0000ffff969fd960 x28: 0000004820001980
[76387.156136] x27: 0000000000e07ef8 x26: 0000000000000000
[76387.156141] x25: 0000ffff9743c6f0 x24: 0000ffff97403000
[76387.156145] x23: 0000000000882608 x22: 0000ffffe67e4baf
[76387.156150] x21: 0000000000000000 x20: 000000000046d970
[76387.156154] x19: 000000000043bf00 x18: 0000000000000a03
[76387.156159] x17: 0000ffff973e6458 x16: 0000004820068eb0
[76387.156163] x15: 0000000000000040 x14: 0000000000000000
[76387.156167] x13: 0000000000000000 x12: 0000000000000008
[76387.156173] x11: 0000000000000000 x10: 0000000000bcaef8
[76387.156186] x9 : 0000004820068fa0 x8 : 000000000000042d
[76387.156198] x7 : 00000000004ed318 x6 : 00000048200687c0
[76387.156203] x5 : 0000000000000000 x4 : 0000000000000000
[76387.156207] x3 : ffffffffffffffff x2 : 0000000000000080
[76387.156212] x1 : 00000048200688b8 x0 : 0000000000000003
[76387.156216]

Huh. Can you test yakkety too? I was fairly sure that it worked, but maybe
i confused myself.
On 2/07/2016 4:01 am, "dann frazier" <email address hidden> wrote:

> With the -proposed version, I'm seeing failures on arm64, apparently due
> to the new dependency on containerd.
>
> Starting Docker Socket for the API.
> Listening on Docker Socket for the API.
> Starting Docker Application Container Engine...
> time="2016-07-01T15:45:38.219004000Z" level=info msg="New containerd
> process, pid: 19074\n"
> time="2016-07-01T15:45:38Z" level=fatal msg="containerd: epoll wait"
> error="function not implemented"
> [76387.156087] containerd[19079]: syscall 1069
> [76387.156096] Code: f94023e4 f94027e5 f9400fe8 d4000001 (b13ffc1f)
> [76387.156102] CPU: 34 PID: 19079 Comm: containerd Tainted: P O
> 4.4.0-29-generic #48
> [76387.156105] Hardware name: Default string Default string/Default
> string, BIOS 5.11 12/12/2012
> [76387.156111] task: ffff801f48b0a700 ti: ffff8019ed864000 task.ti:
> ffff8019ed864000
> [76387.156117] PC is at 0x4ed338
> [76387.156120] LR is at 0x4ed318
> [76387.156123] pc : [<00000000004ed338>] lr : [<00000000004ed318>] pstate:
> 60000000
> [76387.156126] sp : 00000048200687c0
> [76387.156130] x29: 0000ffff969fd960 x28: 0000004820001980
> [76387.156136] x27: 0000000000e07ef8 x26: 0000000000000000
> [76387.156141] x25: 0000ffff9743c6f0 x24: 0000ffff97403000
> [76387.156145] x23: 0000000000882608 x22: 0000ffffe67e4baf
> [76387.156150] x21: 0000000000000000 x20: 000000000046d970
> [76387.156154] x19: 000000000043bf00 x18: 0000000000000a03
> [76387.156159] x17: 0000ffff973e6458 x16: 0000004820068eb0
> [76387.156163] x15: 0000000000000040 x14: 0000000000000000
> [76387.156167] x13: 0000000000000000 x12: 0000000000000008
> [76387.156173] x11: 0000000000000000 x10: 0000000000bcaef8
> [76387.156186] x9 : 0000004820068fa0 x8 : 000000000000042d
> [76387.156198] x7 : 00000000004ed318 x6 : 00000048200687c0
> [76387.156203] x5 : 0000000000000000 x4 : 0000000000000000
> [76387.156207] x3 : ffffffffffffffff x2 : 0000000000000080
> [76387.156212] x1 : 00000048200688b8 x0 : 0000000000000003
> [76387.156216]
>
>
> ** Tags removed: verification-needed
> ** Tags added: verification-failed
>
> --
> You received this bug notification because you are subscribed to
> docker.io in Ubuntu.
> https://bugs.launchpad.net/bugs/1574904
>
> Title:
> Old clients cannot talk to Docker in 16.04
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1574904/+subscriptions
>

Hello <email address hidden>, or anyone else affected,

Accepted containerd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/0.2.1-0ubuntu4~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello <email address hidden>, or anyone else affected,

Accepted docker.io into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io/1.11.2-0ubuntu5~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

it would be useful to give the package a NEWS entry that reminds admins that the daemon needs restarting manually after an upgrade (which will also restart running containers)
Not restarting causes docker info to report itself as 1.11.2 but building containers causes weird errors, like '/bin/sh' not found. They fixed themselves for me after restarting the daemon.

note that in the next docker version 1.12 the restart on update should be handled via the --live-restore flag.

Smoke testing on arm64 looks good - the symptoms in Comment #17 have gone away.

verification-done based on Dann's feedback.

The verification of the Stable Release Update for runc has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package runc - 0.1.1-0ubuntu5~16.04

---------------
runc (0.1.1-0ubuntu5~16.04) xenial; urgency=medium

  * Backport to Xenial. (LP: #1574904)

 -- Michael Hudson-Doyle <email address hidden> Thu, 16 Jun 2016 21:22:26 +1200

This bug was fixed in the package containerd - 0.2.1-0ubuntu4~16.04

---------------
containerd (0.2.1-0ubuntu4~16.04) xenial; urgency=medium

  * Backport to Xenial. (LP: #1574904)

 -- Michael Hudson-Doyle <email address hidden> Thu, 16 Jun 2016 21:19:31 +1200

This bug was fixed in the package docker.io - 1.11.2-0ubuntu5~16.04

---------------
docker.io (1.11.2-0ubuntu5~16.04) xenial; urgency=medium

  * Backport to Xenial. (LP: #1574904)

 -- Michael Hudson-Doyle <email address hidden> Thu, 16 Jun 2016 21:17:51 +1200