LXD Power8 container hangs and slow creation

In case it wasn't clear this is a power 8 system

Can you either use xenial or use the ppa:ubuntu-lxc/stable PPA to get a more recent version of LXC and LXD?

The LXD in wily is pretty ancient and not maintained. We do push security fixes to it but that's the extent of the maintenance we provide for it. We mostly focus on LTS releases so have recent LXD in trusty-backports and in xenial, other releases are better off using the PPA.

will try now

Doug Sikora
Solutions Architect , Canonical Ltd.
+1-848-628-4089

On Thu, Feb 18, 2016 at 3:07 PM, Stéphane Graber <email address hidden>
wrote:

> Can you either use xenial or use the ppa:ubuntu-lxc/stable PPA to get a
> more recent version of LXC and LXD?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1547190
>
> Title:
> LXD functionality on power8
>
> Status in lxc:
> New
>
> Bug description:
> system is a new fresh clean install of Ubuntu Wily ppc64le. Issues
> below are able to be duplicated 100% of the time.
>
> yjetbq@sovmp151:~$ uname -a
> Linux sovmp151 4.2.0-27-generic #32-Ubuntu SMP Fri Jan 22 04:48:03 UTC
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>
> yjetbq@sovmp151:~$ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=15.10
> DISTRIB_CODENAME=wily
> DISTRIB_DESCRIPTION="Ubuntu 15.10"
> yjetbq@sovmp151:~$
>
> Following instructions at https://linuxcontainers.org/lxd/getting-
> started-cli/ I ran into 3 issues as below:
>
> 1st issue: running newgrp lxd does not work:
>
> running newgrp lxd as user prompts for a password, enter password
> and nothing happens, no output from command and user is not added to
> LXD group
>
> Manually added user to LXD group and rebooted.
>
> 2nd issue: container creation takes upward of 3 minutes ., after
> container is created prompt never comes back when trying to get a
> shell via lxc exec <containername> -- /bin/bash . I waited as long as
> 30 minutes no prompt comes back however lxc list shows container
> running with an IP.
>
> steps below
> yjetbq@sovmp151:~$ lxc image list images:
>
> +--------------------------------+--------------+--------+------------------------------------------+---------+----------+------------------------------+
> | ALIAS | FINGERPRINT | PUBLIC |
> DESCRIPTION | ARCH | SIZE | UPLOAD DATE
> |
>
> +--------------------------------+--------------+--------+------------------------------------------+---------+----------+------------------------------+
> | centos/6/amd64 (1 more) | bc1b6c71b6ed | yes | Centos 6
> (amd64) (20160218_02:16) | x86_64 | 49.23MB | Jan 1, 1970 at
> 12:00am (UTC) |
> | centos/6/i386 (1 more) | ec65cfd4be93 | yes | Centos 6
> (i386) (20160218_02:16) | i686 | 49.71MB | Jan 1, 1970 at
> 12:00am (UTC) |
> | centos/7/amd64 (1 more) | 4dfde108d4e0 | yes | Centos 7
> (amd64) (20160218_02:16) | x86_64 | 59.08MB | Jan 1, 1970 at
> 12:00am (UTC) |
> | debian/jessie/amd64 (1 more) | 0d07f11f3f2a | yes | Debian jessie
> (amd64) (20160217_22:42) | x86_64 | 96.88MB | Jan 1, 1970 at 12:00am
> (UTC) |
> | debian/jessie/armel (1 more) | f9e3a6c02f0b | yes | Debian jessie
> (armel) (20160111_22:42) | armv7l | 91.92MB | Jan 1, 1970 at 12:00am
> (UTC) |
> | debian/jessie/armhf (1 more) | 5dad30f84459 | yes | Debian jessie
> (armhf) (20160111_22:42) | armv7l | 92.17MB | Jan 1, 1970 at 12:00am
> (UTC) |
> | debian/jessie/i386 (1 more) | 7020ebc24eb...

SAME ISSUE, however the container did get created faster, well once it did see note at bottom of this comment:

added stable ppa and upgraded (host is on wily):

yjetbq@sovmp151:~$ dpkg -l | grep lxd
ii lxd 2.0.0~beta2-0ubuntu2~ubuntu15.10.1~ppa1 ppc64el Container hypervisor based on LXC - daemon
ii lxd-client 2.0.0~beta2-0ubuntu2~ubuntu15.10.1~ppa1 ppc64el Container hypervisor based on LXC - client
yjetbq@sovmp151:~$ dpkg -l | grep lxc
ii liblxc1 1.1.5-0ubuntu0.15.10.3 ppc64el Linux Containers userspace tools (library)
ii lxc 1.1.5-0ubuntu0.15.10.3 ppc64el Linux Containers userspace tools
ii lxc-templates 2.0.0~beta2-0ubuntu2~ubuntu15.10.1~ppa1 ppc64el Linux Containers userspace tools (templates)
ii lxcfs 2.0.0~beta1-0ubuntu1~ubuntu15.10.1~ppa1 ppc64el FUSE based filesystem for LXC
ii python3-lxc 1.1.5-0ubuntu0.15.10.3 ppc64el Linux Containers userspace tools (Python 3.x bindings)

yjetbq@sovmp151:~$ lxc list
+------------+---------+--------------------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | EPHEMERAL | SNAPSHOTS |
+------------+---------+--------------------------------+------+-----------+-----------+
| ubwilytest | RUNNING | 10.0.3.90 (eth0) | | NO | 0 |
| | | 10.0.4.1 (lxcbr0) | | | |
+------------+---------+--------------------------------+------+-----------+-----------+
yjetbq@sovmp151:~$ lxc stop ubwilytest
yjetbq@sovmp151:~$ lxc delete ubwilytest
yjetbq@sovmp151:~$ time lxc launch ubuntu-wily ubwilytest
Creating ubwilytest
Starting ubwilytest

real 0m47.022s
user 0m0.016s
sys 0m0.004s
yjetbq@sovmp151:~$ lxc exec ubwilytest -- /bin/bash

system hangs here -- no prompt returned

I then created a second container named test - this one took a long time again.

yjetbq@sovmp151:~$ time lxc launch ubuntu-wily test
Creating test
Starting test

real 2m5.821s
user 0m0.008s
sys 0m0.008s
yjetbq@sovmp151:~$

Status changed to 'Confirmed' because the bug affects multiple users.

Move the bug to the right place.

Trying to replicate on our ppc64el system now.

I can reproduce the hang on a 4.2 kernel with latest LXC and LXD.

By the looks of it, this isn't a LXC bug as lxc-attach with the same container works quite successfully. I'm seeing processes getting stuck on a bunch of locks, so might have to do with golang threads somehow, poking at it some more.

Hmm, so nothing actually hangs, it's just the shell behaving very weirdly.

That is, when it looks like it's hung, type for example: "ls -lh /etc" and press enter, and you'll quite successfully get the expected output.

ubuntu@1ss-ppc64el:~/go/src/github.com/lxc/lxd$ lxc exec test -- bash
ls -lh /
total 64K
drwxr-xr-x 2 root root 4.0K Feb 17 15:03 bin
drwxr-xr-x 3 root root 4.0K Feb 17 15:13 boot
drwxr-xr-x 8 root root 460 Feb 19 03:53 dev
drwxr-xr-x 97 root root 4.0K Feb 19 03:49 etc
drwxr-xr-x 3 root root 4.0K Feb 19 03:49 home
drwxr-xr-x 20 root root 4.0K Feb 17 15:13 lib
drwxr-xr-x 2 root root 4.0K Feb 17 15:01 lib64
drwx------ 2 root root 4.0K Feb 17 15:03 lost+found
drwxr-xr-x 2 root root 4.0K Feb 17 15:00 media
drwxr-xr-x 2 root root 4.0K Oct 19 09:14 mnt
drwxr-xr-x 2 root root 4.0K Feb 17 15:00 opt
dr-xr-xr-x 114 nobody nogroup 0 Feb 19 03:53 proc
drwx------ 4 root root 4.0K Feb 19 03:51 root
drwxr-xr-x 17 root root 580 Feb 19 03:53 run
drwxr-xr-x 2 root root 4.0K Feb 17 15:03 sbin
drwxr-xr-x 2 root root 4.0K Feb 17 15:00 srv
dr-xr-xr-x 12 nobody nogroup 0 Feb 19 03:53 sys
drwxrwxrwt 7 root root 4.0K Feb 19 04:17 tmp
drwxr-xr-x 10 root root 4.0K Feb 17 15:00 usr
drwxr-xr-x 12 root root 4.0K Feb 17 15:03 var

Looking a bit more closely, the problem is that the lxd client thinks it's run in non-interactive mode (like reading from a pipe or file). That's because stdin in fact doesn't seem to behave the way we need it to, got to figure out what's special about it on power now...

Right, so the problem is that the ioctl terminal IO arguments differ on ppc64el from x86 and other arches. The Go module we use to deal with this doesn't know any better and so fails miserably.

Rather than use the pure go implementation we have right now (from go crypto) we should re-implement it ourselves to instead use the C library for this.

The right way to fix this issue will be: https://github.com/lxc/lxd/issues/1621

I'll try to get that done upstream tomorrow then once applied upstream, cherry-pick it back into Ubuntu which will then make its way to the PPAs.

Oh and for the slow creation part of this bug, I'm getting container created within 30s on my ppc64el system when using the directory backend which is a normal time.

When enabling btrfs (zfs should be the same), I get a container creation time of ~1s.

ubuntu@1ss-ppc64el:~/go/src/github.com/lxc/lxd$ time lxc launch trusty blah
Starting blah

real 0m0.907s
user 0m0.012s
sys 0m0.000s

This bug was fixed in the package lxd - 2.0.0~beta3-0ubuntu2

---------------
lxd (2.0.0~beta3-0ubuntu2) xenial; urgency=medium

  * Cherry-pick fixes from upstream:
    - Fix static analysis tests
    - Re-implement terminal functions through cgo (LP: #1547190)
    - Move i18n under shared
    - Don't stop containers before asking the user
    - Fix Content-Type value for errors
    - Fix container creation from remote image alias

 -- Stéphane Graber <email address hidden> Fri, 19 Feb 2016 16:31:32 -0500

the container no longer hangs, and i can successfully get a prompt. after creation.

 I still get performance issues though and it is spotty, see output from creating 5 machines below. I can provide you access to this node if need be.

For sanity check:
yjetbq@sovmp151:~$ uname -a
Linux sovmp151 4.2.0-27-generic #32-Ubuntu SMP Fri Jan 22 04:48:03 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
yjetbq@sovmp151:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=15.10
DISTRIB_CODENAME=wily
DISTRIB_DESCRIPTION="Ubuntu 15.10"

yjetbq@sovmp151:~$ dpkg -l | grep lxc || lxd
ii liblxc1 1.1.5-0ubuntu0.15.10.3 ppc64el Linux Containers userspace tools (library)
ii lxc 1.1.5-0ubuntu0.15.10.3 ppc64el Linux Containers userspace tools
ii lxc-templates 2.0.0~beta2-0ubuntu2~ubuntu15.10.1~ppa1 ppc64el Linux Containers userspace tools (templates)
ii lxcfs 2.0.0~beta2-0ubuntu1~ubuntu15.10.1~ppa1 ppc64el FUSE based filesystem for LXC
ii python3-lxc 1.1.5-0ubuntu0.15.10.3 ppc64el Linux Containers userspace tools (Python 3.x bindings)
yjetbq@sovmp151:~$ dpkg -l | grep lxd
ii lxd 2.0.0~beta3-0ubuntu4~ubuntu15.10.1~ppa1 ppc64el Container hypervisor based on LXC - daemon
ii lxd-client 2.0.0~beta3-0ubuntu4~ubuntu15.10.1~ppa1 ppc64el Container hypervisor based on LXC - client

#####################################
Creating machines below as you can see the first 3 machines were OK at less that 58 seconds (not as fast as yours though) then it takes > 2 minutes. I will be kicking off another script and launch a larger number of machines and will share that as well.
######################################

yjetbq@sovmp151:~$ for i in 1 2 3 4 5
> do
> time lxc launch ubuntu-wily ubuntu$i
> done
Creating ubuntu1
Starting ubuntu1

real 0m43.730s
user 0m0.012s
sys 0m0.008s
Creating ubuntu2
Starting ubuntu2

real 0m56.871s
user 0m0.016s
sys 0m0.004s
Creating ubuntu3
Starting ubuntu3

real 0m57.013s
user 0m0.024s
sys 0m0.012s
Creating ubuntu4
Starting ubuntu4

real 2m18.716s
user 0m0.008s
sys 0m0.008s
Creating ubuntu5
Starting ubuntu5

real 2m0.219s
user 0m0.032s
sys 0m0.008s

adding that they actually started:

yjetbq@sovmp151:~$ lxc list
+---------+---------+--------------------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | EPHEMERAL | SNAPSHOTS |
+---------+---------+--------------------------------+------+-----------+-----------+
| ubuntu1 | RUNNING | 10.0.3.6 (eth0) | | NO | 0 |
| | | 10.0.4.1 (lxcbr0) | | | |
+---------+---------+--------------------------------+------+-----------+-----------+
| ubuntu2 | RUNNING | 10.0.3.40 (eth0) | | NO | 0 |
| | | 10.0.4.1 (lxcbr0) | | | |
+---------+---------+--------------------------------+------+-----------+-----------+
| ubuntu3 | RUNNING | 10.0.3.51 (eth0) | | NO | 0 |
| | | 10.0.4.1 (lxcbr0) | | | |
+---------+---------+--------------------------------+------+-----------+-----------+
| ubuntu4 | RUNNING | 10.0.3.165 (eth0) | | NO | 0 |
| | | 10.0.4.1 (lxcbr0) | | | |
+---------+---------+--------------------------------+------+-----------+-----------+
| ubuntu5 | RUNNING | 10.0.3.83 (eth0) | | NO | 0 |
| | | 10.0.4.1 (lxcbr0) | | | |
+---------+---------+--------------------------------+------+-----------+-----------+

That's somewhat expected creation time when not using an optimized storage backend.

Setup zfs, btrfs or LVM for better performance.

On xenial, you should be able to just install "zfsutils-linux" and then run "sudo lxd init" to setup zfs. Note that it requires all existing containers and images be removed first.