adding seccomp rule for socket() fails on i386 since kernel 4.3

Downgrading to https://launchpad.net/ubuntu/+source/systemd/228-2ubuntu1/+build/8333265/+files/systemd-container_228-2ubuntu1_i386.deb fixes this again.

As I suspected, rebuilding the 228-2ubuntu1 systemd source in current xenial does not fix this. The ubuntu1 → ubuntu2 delta was relatively small and does not touch nspawn/seccomp at all. So I figure this is a regression in some -dev package of libc, seccomp, or linux-libc-dev.

Version comparison between the two builds:

 - libseccomp-dev: Both versions built against 2.2.3-2ubuntu1
 - libc6-dev: 2.21-0ubuntu4 → 2.21-0ubuntu5
 - linux-libc-dev: 4.2.0-19.23 → 4.3.0-2.11
 - binutils: 2.25.51.20151113-2ubuntu1 → 2.25.90.20151209-1ubuntu1
 - gcc-5: 5.2.1-24ubuntu3 → 5.3.1-3ubuntu1

This reproduces in a schroot after bind-mounting cgroupfs:

sudo mount -o bind /sys/fs/cgroup/ /var/lib/schroot/mount/schroot-xenial-i386-systemd/sys/fs/cgroup/

I bisected the above toolchain packages, and when building systemd against linux-libc-dev 4.2.0-16.19 it works again.

I tried in the forward direction: linux-libc-dev 4.3.0-4.13 still fails, and that's the latest xenial one (4.3.0-5.14 is not built yet).

I also tried 4.4.0-0.5 in the unstable PPA (https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/unstable/+build/8438036) and it still fails.

For the record: if someone bisects this, I strongly advise to build systemd in a pre-created schroot with

   CFLAGS="-g -O0" DEB_BUILD_FLAGS=nocheck dpkg-buildpackage -Pnoudeb -us -uc -b -j4

which will only take some 3 minutes, instead of 20 .

I further bisected it down to adding this line to /usr/include/i386-linux-gnu/asm/unistd_32.h:

  #define __NR_socket 359

if I drop just that and rebuild systemd, seccomp/nspawn work again.

While systemd does define some syscalls for some more obscure platforms in https://github.com/systemd/systemd/blob/master/src/basic/missing.h if the kernel headers don't already define them, these don't seem to collide with either __NR_socket or the value 359. The only place where I see this referenced is in /usr/include/asm-generic/unistd.h:

#define __NR_socket 198
__SYSCALL(__NR_socket, sys_socket)

but as that redefines __NR_socket I figure that's unrelated. Commenting it out doesn't change the behaviour at all.

I confirm this on Debian sid which has linux-libc-dev 4.3.3-1, exact same situation.

At this point I'm afraid I don't understand what's going on and what these new syscall definitions do.

I tried to #undef __NR_socket in the systemd sources, to see where this value is actually being used. Turns out it is in https://github.com/systemd/systemd/blob/master/src/nspawn/nspawn.c#L1577 in setup_seccomp():

        r = seccomp_rule_add(
                        seccomp,
                        SCMP_ACT_ERRNO(EAFNOSUPPORT),
                        SCMP_SYS(socket),
                        2,
                        SCMP_A0(SCMP_CMP_EQ, AF_NETLINK),
                        SCMP_A2(SCMP_CMP_EQ, NETLINK_AUDIT));
        if (r < 0) {
                log_error_errno(r, "Failed to add audit seccomp rule: %m");

where SCMP_SYS is a macro from libseccomp-dev (/usr/include/seccomp.h):

/**
 * Convert a syscall name into the associated syscall number
 * @param x the syscall name
 */
#define SCMP_SYS(x) (__NR_##x)

So this links the new syscall definition to seccomp. Apparently seccomp_rule_add() (in the same seccomp.h file) behaves differently if the syscall is defined. I just wonder how this actually built on i386 with the 4.2.0 kernel headers which did not have __NR_socket defined?

With current 4.3 kernel headers, the value of SCMP_SYS(socket) == 359, as defined above. With the previous 4.2 kernel headers, the value is 4294967195 == 0xFFFFFF9B instead, apparently some auto-generated value. So this explains how it built before.

So it looks like this might be between libseccomp and the kernel now?

I now isolated this seccomp failure into a tiny .c file which reproduces this. On amd64 it works:

$ gcc -o /tmp/o ~/seccomp-socket-filter.c -lseccomp && /tmp/o
SCMP_SYS(socket) == 41 == 29
Success

and on i386 it reproduces the error:

$ gcc -o /tmp/o ~/seccomp-socket-filter.c -lseccomp && /tmp/o
SCMP_SYS(socket) == 359 == 167
seccomp_rule_add failed: Bad address

So what systemd is trying to do is to first initialize seccomp with possible alternative architectures (running 32 bit container on 64 bit host, and vice versa if you have a 64 bit kernel) and then disallow opening socket()s to the netlink audit subsystem, as audit is broken for containers. The gist of it is

    seccomp = seccomp_init(SCMP_ACT_ALLOW);
    seccomp_arch_add(seccomp, SCMP_ARCH_X86_64);
    seccomp_rule_add(
            seccomp,
            SCMP_ACT_ERRNO(EAFNOSUPPORT),
            SCMP_SYS(socket),
            2,
            SCMP_A0(SCMP_CMP_EQ, AF_NETLINK),
            SCMP_A2(SCMP_CMP_EQ, NETLINK_AUDIT));

This has worked on both arches until __NR_socket got defined on i386, before it used that autogenerated value.

This isn't specific to netlink. I removed the two rules from the seccomp filter and simplified it to just generally block socket(). I also simplified adding the arches so that only the non-native arch is added, not the native one. Note that adding the socket() filter *does* work on both arches if the non-native architecture does not get added, this only fails with adding x86_64 to the filter on i386.

Forgot to attach the simplified file..

So in the commit below we switched how the socket family of calls are exposed at the syscall level (which was a 4.3-rc1 change):

  commit 9dea5dc921b5f4045a18c63eb92e84dc274d17eb
  Author: Andy Lutomirski <email address hidden>
  Date: Tue Jul 14 15:24:24 2015 -0700

    x86/entry/syscalls: Wire up 32-bit direct socket calls

One of the stated goals of this was to expose these calls for seccomp mediation and to bring 32bit in line with 64bit. So it is cirtain we never did do seccomp mediation on these before.

Notified systemd upstream in https://github.com/systemd/systemd/issues/2177 .

Running the example above the EFAULT is being generated in userspace. Looking at libseccomp it seems we have a literal copy of the systemcall table mapping call strings to local numbers. For 32bit the new system calls are not filled in so they will fail. Esentially libseccomp and the kernel headers are out of sync, so systemd thinks it can use real mitigation on socket() but libseccomp does not think 32bit supports it.

https://github.com/seccomp/libseccomp/pull/22 Thanks Andy!

This bug was fixed in the package libseccomp - 2.2.3-2ubuntu3

---------------
libseccomp (2.2.3-2ubuntu3) xenial; urgency=low

  * debian/patches/add-x86-32bit-socket-calls.patch: add the newly
    connected direct socket calls. (LP: #1526358)

 -- Andy Whitcroft <email address hidden> Wed, 16 Dec 2015 14:30:17 +0000