Docker can't start s390x images on Ubuntu and gets "oci runtime error: unrecognized architecture"

I was able to recreate this problem ... (only with 1.12.3, not with 1.12.1)

Regression is in runc rc1 vs r2, out of date libseccomp vedorised golang dependency which (artificially) limited execution on s390x.

This is resolved in zesty-proposed, investigating why it has not migrated to zesty-release yet.

Following that will investigate to cherry-pick runc fix-up or expedite rc2 backport SRU.

Also affects ArchPPC64LE

Starting bileto landings for this SRU for yakkety and xenial:

Yakkety: https://bileto.ubuntu.com/#/ticket/2388

Xenial: https://bileto.ubuntu.com/#/ticket/2389

This is resolved in zesty-proposed, however those packages have not migrated yet to zesty-release due to a regression of running docker-in-lxd test case.

Hello bugproxy, or anyone else affected,

Accepted runc into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.0.0~rc1-0ubuntu2~16.10.1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello bugproxy, or anyone else affected,

Accepted runc into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.0.0~rc1-0ubuntu2~16.04.1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I was able to verify that runc 1.0.0~rc1-0ubuntu2~16.04.1.1 (from xenial.proposed)
with docker.io 1.12.3-0ubuntu4~16.04.2 makes docker working again.

See below:

ubuntu@hwe0010:~$ apt-cache policy runc
runc:
  Installed: 1.0.0~rc1-0ubuntu2~16.04.1.1
  Candidate: 1.0.0~rc1-0ubuntu2~16.04.1.1
  Version table:
 *** 1.0.0~rc1-0ubuntu2~16.04.1.1 500
        500 http://ports.ubuntu.com/ubuntu-ports xenial-proposed/universe s390x Packages
        100 /var/lib/dpkg/status
     1.0.0~rc1-0ubuntu2~16.04.1 500
        500 http://us.ports.ubuntu.com/ubuntu-ports xenial-updates/universe s390x Packages
     0.0.8+dfsg-2 500
        500 http://us.ports.ubuntu.com/ubuntu-ports xenial/universe s390x Packages
ubuntu@hwe0010:~$ apt-cache policy docker.io
docker.io:
  Installed: 1.12.3-0ubuntu4~16.04.2
  Candidate: 1.12.3-0ubuntu4~16.04.2
  Version table:
 *** 1.12.3-0ubuntu4~16.04.2 500
        500 http://us.ports.ubuntu.com/ubuntu-ports xenial-updates/universe s390x Packages
        100 /var/lib/dpkg/status
     1.10.3-0ubuntu6 500
        500 http://us.ports.ubuntu.com/ubuntu-ports xenial/universe s390x Packages
ubuntu@hwe0010:~$ docker run -it s390x/ubuntu bash
root@de34f1271c3d:/# ls -la .dockerenv
-rwxr-xr-x 1 root root 0 Jan 20 16:18 .dockerenv
root@de34f1271c3d:/# exit
exit
ubuntu@hwe0010:~$

I could also verify on yakkety - please see below, incl. negative test (had to restart docker after the update to 1.12.3):

ubuntu@s1lp15:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.10
Release: 16.10
Codename: yakkety
ubuntu@s1lp15:~$ apt-cache policy docker.io
docker.io:
  Installed: 1.12.1-0ubuntu15
  Candidate: 1.12.3-0ubuntu4~16.10.2
  Version table:
     1.12.3-0ubuntu4~16.10.2 500
        500 http://ports.ubuntu.com yakkety-updates/universe s390x Packages
 *** 1.12.1-0ubuntu15 500
        500 http://ports.ubuntu.com yakkety/universe s390x Packages
        100 /var/lib/dpkg/status
ubuntu@s1lp15:~$ docker run -it s390x/ubuntu bash
Unable to find image 's390x/ubuntu:latest' locally
latest: Pulling from s390x/ubuntu
b43bc799d4e2: Pull complete
afbd2b8f99bf: Pull complete
0e3aee0ec255: Pull complete
eb19a6534950: Pull complete
29583cd8abac: Pull complete
Digest: sha256:73e317083b0e7f6f477500b97f658519448aa7a54842a077ae8465235d6b8598
Status: Downloaded newer image for s390x/ubuntu:latest
root@2362540d5f1b:/#
root@2362540d5f1b:/# exit
exit
ubuntu@s1lp15:~$

ubuntu@s1lp15:~$ sudo apt install docker.io
...
ubuntu@s1lp15:~$ apt-cache policy docker.io
docker.io:
  Installed: 1.12.3-0ubuntu4~16.10.2
  Candidate: 1.12.3-0ubuntu4~16.10.2
  Version table:
 *** 1.12.3-0ubuntu4~16.10.2 500
        500 http://ports.ubuntu.com yakkety-updates/universe s390x Packages
        100 /var/lib/dpkg/status
     1.12.1-0ubuntu15 500
        500 http://ports.ubuntu.com yakkety/universe s390x Packages
ubuntu@s1lp15:~$ docker run -it s390x/ubuntu bash
docker: Error response from daemon: oci runtime error: process_linux.go:330: running prestart hook 0 caused "fork/exec /usr/bin/dockerd (deleted): no such file or directory: ".
ubuntu@s1lp15:~$ apt-cache policy runc
runc:
  Installed: 1.0.0~rc1-0ubuntu2~16.10.1.1
  Candidate: 1.0.0~rc1-0ubuntu2~16.10.1.1
  Version table:
 *** 1.0.0~rc1-0ubuntu2~16.10.1.1 500
        500 http://ports.ubuntu.com yakkety-proposed/universe s390x Packages
        100 /var/lib/dpkg/status
     1.0.0~rc1-0ubuntu2~16.10.1 500
        500 http://ports.ubuntu.com yakkety-updates/universe s390x Packages
     1.0.0~rc1-0ubuntu1 500
        500 http://ports.ubuntu.com yakkety/universe s390x Packages
ubuntu@s1lp15:~$ docker run -it s390x/ubuntu bash
docker: Error response from daemon: oci runtime error: process_linux.go:330: running prestart hook 0 caused "fork/exec /usr/bin/dockerd (deleted): no such file or directory: ".
ubuntu@s1lp15:~$ dpkg -l runc
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii runc 1.0.0~rc1-0u s390x Open Container Project - runtime
ubuntu@s1lp15:~$ sudo systemctl restart docker
ubuntu@s1lp15:~$ docker run -it s390x/ubuntu bash
root@22c6dfde2ec7:/# ls -la .dockerenv
-rwxr-xr-x 1 root root 0 Jan 20 16:29 .dockerenv
root@22c6dfde2ec7:/# exit
exit
ubuntu@s1lp15:~$

verified on ppc64el xenial; s390x xenial & yakkety; amd64 xenial.

This bug was fixed in the package runc - 1.0.0~rc1-0ubuntu2~16.04.1.1

---------------
runc (1.0.0~rc1-0ubuntu2~16.04.1.1) xenial; urgency=medium

  * Cherrypick seccomp update from -rc2 to resolve failure to execute on
    ppc64el and s390x. LP: #1658009

 -- Dimitri John Ledkov <email address hidden> Fri, 20 Jan 2017 11:09:00 +0000

The verification of the Stable Release Update for runc has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package runc - 1.0.0~rc1-0ubuntu2~16.10.1.1

---------------
runc (1.0.0~rc1-0ubuntu2~16.10.1.1) yakkety; urgency=medium

  * Cherrypick seccomp update from -rc2 to resolve failure to execute on
    ppc64el and s390x. LP: #1658009

 -- Dimitri John Ledkov <email address hidden> Fri, 20 Jan 2017 11:06:50 +0000

Oh nuts, I dropped the ball on verifying platforms that don't have
autopkgtests :-( I'm on leave for a few days but catch me on telegram if
you want to talk about fixes?

sent from my phone, please excuse brevity

On 20 Jan 2017 23:25, "Dimitri John Ledkov" <email address hidden> wrote:

> Regression is in runc rc1 vs r2, out of date libseccomp vedorised golang
> dependency which (artificially) limited execution on s390x.
>
> This is resolved in zesty-proposed, investigating why it has not
> migrated to zesty-release yet.
>
> Following that will investigate to cherry-pick runc fix-up or expedite
> rc2 backport SRU.
>
> ** Package changed: docker.io (Ubuntu Xenial) => runc (Ubuntu Xenial)
>
> --
> You received this bug notification because you are subscribed to
> docker.io in Ubuntu.
> https://bugs.launchpad.net/bugs/1658009
>
> Title:
> Docker can't start s390x images on Ubuntu and gets "oci runtime error:
> unrecognized architecture"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu-z-systems/+bug/1658009/+subscriptions
>

------- Comment From <email address hidden> 2017-01-23 10:16 EDT-------
Fixed on xenial and yakkety with the new runc package.
IBM bugzilla -> closed

New enough runc is in zesty now as well. Closing ticket.