Bug #1307215 “destroy-environment fails to clear lxc containers” : Bugs : juju-core

Revision history for this message

Tim Penhey (thumper) wrote on 2014-04-14:

#1

Hi Evan, Can I get you to please run and pastebin the following?

juju destroy-environment local --logging-config=golxc=TRACE;juju=DEBUG --show-log

This will give us the output of the lxc command and why it is failing.

Curtis Hovey (sinzui) on 2014-04-14

tags:	added: destroy-environment local-provider lxc
Changed in juju-core:
status:	New → Triaged
importance:	Undecided → High
milestone:	none → 1.20.0

Revision history for this message

Evan (ev) wrote on 2014-04-17:

#2

I've been unable to reproduce this thus far, but I'll keep at it.

Revision history for this message

John A Meinel (jameinel) wrote on 2014-04-24:

#3

Until we can reproduce this, I don't think we can address it.

Changed in juju-core:
status:	Triaged → Incomplete

Curtis Hovey (sinzui) on 2014-05-01

Changed in juju-core:
milestone:	1.20.0 → none

Curtis Hovey (sinzui) on 2014-05-12

Changed in juju-core:
importance:	High → Medium

Revision history for this message

Launchpad Janitor (janitor) wrote on 2014-07-12:

#4

[Expired for juju-core because there has been no activity for 60 days.]

Changed in juju-core:
status:	Incomplete → Expired

Revision history for this message

Evan (ev) wrote on 2014-07-12:

#5

Download full text (33.6 KiB)

I've had this happen again. It looks like it lxc cannot remove the rootfs subvolume because it references other subvolumes:
http://lxr.free-electrons.com/source/fs/btrfs/ioctl.c#L1894

(.venv-ubuntu)vagrant@vagrant-ubuntu-trusty-64:/ev/bzr/uci-engine/ceph$ sudo strace -e file lxc-destroy --force --logpriority=DEBUG --name vagrant-local-machine-1
execve("/usr/bin/lxc-destroy", ["lxc-destroy", "--force", "--logpriority=DEBUG", "--name", "vagrant-local-machine-1"], [/* 16 vars */]) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/liblxc.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libapparmor.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libseccomp.so.2", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcgmanager.so.0", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libnih.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libnih-dbus.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdbus-1.so.3", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
statfs("/sys/fs/selinux", 0x7fffe99765f0) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7fffe99765f0) = -1 ENOENT (No such file or directory)
open("/proc/filesystems", O_RDONLY) = 3
open("/proc/cgroups", O_RDONLY|O_CLOEXEC) = 3
stat("/sys/kernel/se...

I've had this happen again. It looks like it lxc cannot remove the rootfs subvolume because it references other subvolumes:
http://lxr.free-electrons.com/source/fs/btrfs/ioctl.c#L1894

(.venv-ubuntu)vagrant@vagrant-ubuntu-trusty-64:/ev/bzr/uci-engine/ceph$ sudo strace -e file lxc-destroy --force --logpriority=DEBUG --name vagrant-local-machine-1
execve("/usr/bin/lxc-destroy", ["lxc-destroy", "--force", "--logpriority=DEBUG", "--name", "vagrant-local-machine-1"], [/* 16 vars */]) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/liblxc.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libapparmor.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libseccomp.so.2", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcgmanager.so.0", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libnih.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libnih-dbus.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdbus-1.so.3", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
statfs("/sys/fs/selinux", 0x7fffe99765f0) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7fffe99765f0)      = -1 ENOENT (No such file or directory)
open("/proc/filesystems", O_RDONLY)     = 3
open("/proc/cgroups", O_RDONLY|O_CLOEXEC) = 3
stat("/sys/kernel/security/apparmor/features/mount/mask", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
open("/sys/module/apparmor/parameters/enabled", O_RDONLY) = 3
open("/etc/lxc/lxc.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
mkdir("/", 0755)                        = -1 EEXIST (File exists)
mkdir("/run/", 0755)                    = -1 EEXIST (File exists)
mkdir("/run/lock/", 0755)               = -1 EEXIST (File exists)
mkdir("/run/lock/lxc//", 0755)          = -1 EEXIST (File exists)
mkdir("/run/lock/lxc//var/", 0755)      = -1 EEXIST (File exists)
mkdir("/run/lock/lxc//var/lib/", 0755)  = -1 EEXIST (File exists)
mkdir("/run/lock/lxc//var/lib/lxc", 0755) = -1 EEXIST (File exists)
stat("/var/lib/lxc/vagrant-local-machine-1/config", {st_mode=S_IFREG|0644, st_size=1763, ...}) = 0
open("/run/lock/lxc//var/lib/lxc/vagrant-local-machine-1", O_RDWR|O_CREAT, 0600) = 3
access("/var/lib/lxc/vagrant-local-machine-1/config", R_OK) = 0
open("/var/lib/lxc/vagrant-local-machine-1/config", O_RDONLY) = 4
open("/dev/urandom", O_RDONLY)          = 5
stat("/var/lib/lxc/vagrant-local-machine-1/partial", 0x7fffe9976500) = -1 ENOENT (No such file or directory)
stat("/var/lib/lxc/vagrant-local-machine-1/config", {st_mode=S_IFREG|0644, st_size=1763, ...}) = 0
stat("/var/lib/lxc/vagrant-local-machine-1/config", {st_mode=S_IFREG|0644, st_size=1763, ...}) = 0
open("/run/lock/lxc//var/lib/lxc/vagrant-local-machine-1", O_RDWR|O_CREAT, 0600) = 3
open("/var/lib/lxc/vagrant-local-machine-1/lxc_snapshots", O_RDONLY) = -1 ENOENT (No such file or directory)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19557, si_status=127, si_utime=0, si_stime=0} ---
stat("/var/lib/lxc/vagrant-local-machine-1/rootfs", {st_mode=S_IFDIR|0755, st_size=152, ...}) = 0
open("/var/lib/lxc/vagrant-local-machine-1/rootfs", O_RDONLY) = 4
stat("/var/lib/lxc/vagrant-local-machine-1/rootfs", {st_mode=S_IFDIR|0755, st_size=152, ...}) = 0
open("/var/lib/lxc/vagrant-local-machine-1", O_RDONLY) = 4
lxc_container: Error destroying rootfs for vagrant-local-machine-1
Destroying vagrant-local-machine-1 failed
+++ exited with 1 +++
(.venv-ubuntu)vagrant@vagrant-ubuntu-trusty-64:/ev/bzr/uci-engine/ceph$ sudo strace lxc-destroy --force --logpriority=DEBUG --name vagrant-local-machine-1
execve("/usr/bin/lxc-destroy", ["lxc-destroy", "--force", "--logpriority=DEBUG", "--name", "vagrant-local-machine-1"], [/* 16 vars */]) = 0
brk(0)                                  = 0x7fb57f8e2000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eacd000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=30455, ...}) = 0
mmap(NULL, 30455, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb57eac5000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/liblxc.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\306\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=431024, ...}) = 0
mmap(NULL, 2526312, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57e644000
mprotect(0x7fb57e6aa000, 2097152, PROT_NONE) = 0
mmap(0x7fb57e8aa000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x66000) = 0x7fb57e8aa000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\37\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1845024, ...}) = 0
mmap(NULL, 3953344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57e27e000
mprotect(0x7fb57e43a000, 2093056, PROT_NONE) = 0
mmap(0x7fb57e639000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bb000) = 0x7fb57e639000
mmap(0x7fb57e63f000, 17088, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb57e63f000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \26\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=18952, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eac4000
mmap(NULL, 2114160, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57e079000
mprotect(0x7fb57e07d000, 2093056, PROT_NONE) = 0
mmap(0x7fb57e27c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7fb57e27c000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libapparmor.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\30\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=47984, ...}) = 0
mmap(NULL, 2143152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57de6d000
mprotect(0x7fb57de78000, 2093056, PROT_NONE) = 0
mmap(0x7fb57e077000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x7fb57e077000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libseccomp.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\207\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=92336, ...}) = 0
mmap(NULL, 2187464, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57dc56000
mprotect(0x7fb57dc67000, 2097152, PROT_NONE) = 0
mmap(0x7fb57de67000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11000) = 0x7fb57de67000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcgmanager.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0206\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=108480, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eac3000
mmap(NULL, 2203688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57da3b000
mprotect(0x7fb57da54000, 2093056, PROT_NONE) = 0
mmap(0x7fb57dc53000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7fb57dc53000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libnih.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`G\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=96280, ...}) = 0
mmap(NULL, 2191776, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57d823000
mprotect(0x7fb57d83a000, 2093056, PROT_NONE) = 0
mmap(0x7fb57da39000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7fb57da39000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libnih-dbus.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340,\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=38920, ...}) = 0
mmap(NULL, 2134040, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57d619000
mprotect(0x7fb57d621000, 2097152, PROT_NONE) = 0
mmap(0x7fb57d821000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7fb57d821000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdbus-1.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@h\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=281552, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eac2000
mmap(NULL, 2377408, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57d3d4000
mprotect(0x7fb57d418000, 2093056, PROT_NONE) = 0
mmap(0x7fb57d617000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x43000) = 0x7fb57d617000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\17\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=10680, ...}) = 0
mmap(NULL, 2105624, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57d1d1000
mprotect(0x7fb57d1d3000, 2093056, PROT_NONE) = 0
mmap(0x7fb57d3d2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7fb57d3d2000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0po\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=141574, ...}) = 0
mmap(NULL, 2217264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57cfb3000
mprotect(0x7fb57cfcc000, 2093056, PROT_NONE) = 0
mmap(0x7fb57d1cb000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7fb57d1cb000
mmap(0x7fb57d1cd000, 13616, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb57d1cd000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0[\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=134296, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eac1000
mmap(NULL, 2238192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57cd90000
mprotect(0x7fb57cdb0000, 2093056, PROT_NONE) = 0
mmap(0x7fb57cfaf000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1f000) = 0x7fb57cfaf000
mmap(0x7fb57cfb1000, 5872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb57cfb1000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P#\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=31792, ...}) = 0
mmap(NULL, 2129016, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57cb88000
mprotect(0x7fb57cb8f000, 2093056, PROT_NONE) = 0
mmap(0x7fb57cd8e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7fb57cd8e000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\27\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=252032, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eac0000
mmap(NULL, 2347200, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57c94a000
mprotect(0x7fb57c987000, 2093056, PROT_NONE) = 0
mmap(0x7fb57cb86000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3c000) = 0x7fb57cb86000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14664, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb57c746000
mprotect(0x7fb57c749000, 2093056, PROT_NONE) = 0
mmap(0x7fb57c948000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fb57c948000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eabf000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eabe000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eabc000
arch_prctl(ARCH_SET_FS, 0x7fb57eabc8c0) = 0
mprotect(0x7fb57e639000, 16384, PROT_READ) = 0
mprotect(0x7fb57c948000, 4096, PROT_READ) = 0
mprotect(0x7fb57cb86000, 4096, PROT_READ) = 0
mprotect(0x7fb57d1cb000, 4096, PROT_READ) = 0
mprotect(0x7fb57cd8e000, 4096, PROT_READ) = 0
mprotect(0x7fb57cfaf000, 4096, PROT_READ) = 0
mprotect(0x7fb57d3d2000, 4096, PROT_READ) = 0
mprotect(0x7fb57d617000, 4096, PROT_READ) = 0
mprotect(0x7fb57da39000, 4096, PROT_READ) = 0
mprotect(0x7fb57d821000, 4096, PROT_READ) = 0
mprotect(0x7fb57dc53000, 8192, PROT_READ) = 0
mprotect(0x7fb57de67000, 20480, PROT_READ) = 0
mprotect(0x7fb57e077000, 4096, PROT_READ) = 0
mprotect(0x7fb57e27c000, 4096, PROT_READ) = 0
mprotect(0x7fb57e8aa000, 8192, PROT_READ) = 0
mprotect(0x7fb57ecd3000, 4096, PROT_READ) = 0
mprotect(0x7fb57eacf000, 4096, PROT_READ) = 0
munmap(0x7fb57eac5000, 30455)           = 0
set_tid_address(0x7fb57eabcb90)         = 19562
set_robust_list(0x7fb57eabcba0, 24)     = 0
futex(0x7fff1ecaea60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7fb57eabc8c0) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0x7fb57cfb99f0, [], SA_RESTORER|SA_SIGINFO, 0x7fb57cfc3340}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7fb57cfb9a80, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fb57cfc3340}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x7fff1ecaea30) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7fff1ecaea30)      = -1 ENOENT (No such file or directory)
brk(0)                                  = 0x7fb57f8e2000
brk(0x7fb57f903000)                     = 0x7fb57f903000
open("/proc/filesystems", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eacc000
read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tr"..., 1024) = 336
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7fb57eacc000, 4096)            = 0
open("/proc/cgroups", O_RDONLY|O_CLOEXEC) = 3
fcntl(3, F_GETFL)                       = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eacc000
lseek(3, 0, SEEK_CUR)                   = 0
read(3, "#subsys_name\thierarchy\tnum_cgrou"..., 1024) = 166
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7fb57eacc000, 4096)            = 0
clock_getres(CLOCK_MONOTONIC, {0, 1})   = 0
getresuid([0], [0], [0])                = 0
getresgid([0], [0], [0])                = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_LOCAL, sun_path="/sys/fs/cgroup/cgmanager/sock"}, 31) = 0
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
geteuid()                               = 0
getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0
poll([{fd=3, events=POLLOUT}], 1, 0)    = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "\0", 1, MSG_NOSIGNAL, NULL, 0) = 1
sendto(3, "AUTH EXTERNAL 30\r\n", 18, MSG_NOSIGNAL, NULL, 0) = 18
poll([{fd=3, events=POLLIN}], 1, 4294967295) = 1 ([{fd=3, revents=POLLIN}])
read(3, "OK e24294b19eccf4df2f67b79b53c06"..., 2048) = 37
poll([{fd=3, events=POLLOUT}], 1, 4294967295) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "NEGOTIATE_UNIX_FD\r\n", 19, MSG_NOSIGNAL, NULL, 0) = 19
poll([{fd=3, events=POLLIN}], 1, 4294967295) = 1 ([{fd=3, revents=POLLIN}])
read(3, "AGREE_UNIX_FD\r\n", 2048)      = 15
poll([{fd=3, events=POLLOUT}], 1, 4294967295) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "BEGIN\r\n", 7, MSG_NOSIGNAL, NULL, 0) = 7
poll([{fd=3, events=POLLIN|POLLOUT}], 1, 4294967295) = 1 ([{fd=3, revents=POLLOUT}])
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\0\1\4\0\0\0\1\0\0\0o\0\0\0\1\1o\0\36\0\0\0/org/lin"..., 128}, {"\0\0\0\0", 4}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 132
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\0\0\0\0\1\0\0\0\10\0\0\0\5\1u\0\1\0\0\0", 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 24
recvmsg(3, 0x7fff1ecae6e0, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
geteuid()                               = 0
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\0\1\30\0\0\0\2\0\0\0y\0\0\0\1\1o\0\36\0\0\0/org/lin"..., 144}, {"\6\0\0\0cpuset\0\0\1\0\0\0/\0\0\0jL\0\0", 24}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 168
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\0\0\0\0\2\0\0\0\10\0\0\0\5\1u\0\2\0\0\0", 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 24
recvmsg(3, 0x7fff1ecae720, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\0\1\24\0\0\0\3\0\0\0y\0\0\0\1\1o\0\36\0\0\0/org/lin"..., 144}, {"\3\0\0\0cpu\0\1\0\0\0/\0\0\0jL\0\0", 20}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 164
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\0\0\0\0\3\0\0\0\10\0\0\0\5\1u\0\3\0\0\0", 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 24
recvmsg(3, 0x7fff1ecae720, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\0\1\30\0\0\0\4\0\0\0y\0\0\0\1\1o\0\36\0\0\0/org/lin"..., 144}, {"\7\0\0\0cpuacct\0\1\0\0\0/\0\0\0jL\0\0", 24}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 168
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\0\0\0\0\4\0\0\0\10\0\0\0\5\1u\0\4\0\0\0", 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 24
recvmsg(3, 0x7fff1ecae720, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\0\1\30\0\0\0\5\0\0\0y\0\0\0\1\1o\0\36\0\0\0/org/lin"..., 144}, {"\6\0\0\0memory\0\0\1\0\0\0/\0\0\0jL\0\0", 24}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 168
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\0\0\0\0\5\0\0\0\10\0\0\0\5\1u\0\5\0\0\0", 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 24
recvmsg(3, 0x7fff1ecae720, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\0\1\30\0\0\0\6\0\0\0y\0\0\0\1\1o\0\36\0\0\0/org/lin"..., 144}, {"\7\0\0\0devices\0\1\0\0\0/\0\0\0jL\0\0", 24}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 168
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\0\0\0\0\6\0\0\0\10\0\0\0\5\1u\0\6\0\0\0", 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 24
recvmsg(3, 0x7fff1ecae720, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\0\1\30\0\0\0\7\0\0\0y\0\0\0\1\1o\0\36\0\0\0/org/lin"..., 144}, {"\7\0\0\0freezer\0\1\0\0\0/\0\0\0jL\0\0", 24}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 168
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\0\0\0\0\7\0\0\0\10\0\0\0\5\1u\0\7\0\0\0", 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 24
recvmsg(3, 0x7fff1ecae720, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\0\1\30\0\0\0\10\0\0\0y\0\0\0\1\1o\0\36\0\0\0/org/lin"..., 144}, {"\5\0\0\0blkio\0\0\0\1\0\0\0/\0\0\0jL\0\0", 24}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 168
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\0\0\0\0\10\0\0\0\10\0\0\0\5\1u\0\10\0\0\0", 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 24
recvmsg(3, 0x7fff1ecae720, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\0\1\34\0\0\0\t\0\0\0y\0\0\0\1\1o\0\36\0\0\0/org/lin"..., 144}, {"\n\0\0\0perf_event\0\0\1\0\0\0/\0\0\0jL\0\0", 28}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 172
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\0\0\0\0\t\0\0\0\10\0\0\0\5\1u\0\t\0\0\0", 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 24
recvmsg(3, 0x7fff1ecae720, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\0\1\30\0\0\0\n\0\0\0y\0\0\0\1\1o\0\36\0\0\0/org/lin"..., 144}, {"\7\0\0\0hugetlb\0\1\0\0\0/\0\0\0jL\0\0", 24}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 168
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\0\0\0\0\n\0\0\0\10\0\0\0\5\1u\0\n\0\0\0", 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 24
recvmsg(3, 0x7fff1ecae720, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
close(3)                                = 0
stat("/sys/kernel/security/apparmor/features/mount/mask", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
open("/sys/module/apparmor/parameters/enabled", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eacc000
read(3, "Y\n", 4096)                    = 2
close(3)                                = 0
munmap(0x7fb57eacc000, 4096)            = 0
geteuid()                               = 0
open("/etc/lxc/lxc.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
geteuid()                               = 0
mkdir("/", 0755)                        = -1 EEXIST (File exists)
mkdir("/run/", 0755)                    = -1 EEXIST (File exists)
mkdir("/run/lock/", 0755)               = -1 EEXIST (File exists)
mkdir("/run/lock/lxc//", 0755)          = -1 EEXIST (File exists)
mkdir("/run/lock/lxc//var/", 0755)      = -1 EEXIST (File exists)
mkdir("/run/lock/lxc//var/lib/", 0755)  = -1 EEXIST (File exists)
mkdir("/run/lock/lxc//var/lib/lxc", 0755) = -1 EEXIST (File exists)
stat("/var/lib/lxc/vagrant-local-machine-1/config", {st_mode=S_IFREG|0644, st_size=1763, ...}) = 0
open("/run/lock/lxc//var/lib/lxc/vagrant-local-machine-1", O_RDWR|O_CREAT, 0600) = 3
fcntl(3, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0
access("/var/lib/lxc/vagrant-local-machine-1/config", R_OK) = 0
open("/var/lib/lxc/vagrant-local-machine-1/config", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=1763, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eacc000
read(4, "lxc.mount = /var/lib/lxc/vagrant"..., 4096) = 1763
open("/dev/urandom", O_RDONLY)          = 5
fstat(5, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff1ecae690) = -1 EINVAL (Invalid argument)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eacb000
read(5, "\237\370;\37\352d\253\364\327gkW\225\227x\3\220A\277 b^\376`a\10\277q\261)\nS"..., 4096) = 4096
close(5)                                = 0
munmap(0x7fb57eacb000, 4096)            = 0
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x7fb57eacc000, 4096)            = 0
fcntl(3, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
close(3)                                = 0
stat("/var/lib/lxc/vagrant-local-machine-1/partial", 0x7fff1ecae940) = -1 ENOENT (No such file or directory)
socket(PF_LOCAL, SOCK_STREAM, 0)        = 3
connect(3, {sa_family=AF_LOCAL, sun_path=@"/var/lib/lxc/vagrant-local-machine-1/command"}, 47) = -1 ECONNREFUSED (Connection refused)
connect(3, {sa_family=AF_LOCAL, sun_path=@"/var/lib/lxc/vagrant-local-machine-1/command"}, 110) = -1 ECONNREFUSED (Connection refused)
close(3)                                = 0
stat("/var/lib/lxc/vagrant-local-machine-1/config", {st_mode=S_IFREG|0644, st_size=1763, ...}) = 0
socket(PF_LOCAL, SOCK_STREAM, 0)        = 3
connect(3, {sa_family=AF_LOCAL, sun_path=@"/var/lib/lxc/vagrant-local-machine-1/command"}, 47) = -1 ECONNREFUSED (Connection refused)
connect(3, {sa_family=AF_LOCAL, sun_path=@"/var/lib/lxc/vagrant-local-machine-1/command"}, 110) = -1 ECONNREFUSED (Connection refused)
close(3)                                = 0
socket(PF_LOCAL, SOCK_STREAM, 0)        = 3
connect(3, {sa_family=AF_LOCAL, sun_path=@"/var/lib/lxc/vagrant-local-machine-1/command"}, 47) = -1 ECONNREFUSED (Connection refused)
connect(3, {sa_family=AF_LOCAL, sun_path=@"/var/lib/lxc/vagrant-local-machine-1/command"}, 110) = -1 ECONNREFUSED (Connection refused)
close(3)                                = 0
stat("/var/lib/lxc/vagrant-local-machine-1/config", {st_mode=S_IFREG|0644, st_size=1763, ...}) = 0
open("/run/lock/lxc//var/lib/lxc/vagrant-local-machine-1", O_RDWR|O_CREAT, 0600) = 3
fcntl(3, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0
socket(PF_LOCAL, SOCK_STREAM, 0)        = 4
connect(4, {sa_family=AF_LOCAL, sun_path=@"/var/lib/lxc/vagrant-local-machine-1/command"}, 47) = -1 ECONNREFUSED (Connection refused)
connect(4, {sa_family=AF_LOCAL, sun_path=@"/var/lib/lxc/vagrant-local-machine-1/command"}, 110) = -1 ECONNREFUSED (Connection refused)
close(4)                                = 0
socket(PF_LOCAL, SOCK_STREAM, 0)        = 4
connect(4, {sa_family=AF_LOCAL, sun_path=@"/var/lib/lxc/vagrant-local-machine-1/command"}, 47) = -1 ECONNREFUSED (Connection refused)
connect(4, {sa_family=AF_LOCAL, sun_path=@"/var/lib/lxc/vagrant-local-machine-1/command"}, 110) = -1 ECONNREFUSED (Connection refused)
close(4)                                = 0
open("/var/lib/lxc/vagrant-local-machine-1/lxc_snapshots", O_RDONLY) = -1 ENOENT (No such file or directory)
geteuid()                               = 0
pipe2([4, 5], O_CLOEXEC)                = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb57eabcb90) = 19563
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19563, si_status=127, si_utime=0, si_stime=0} ---
close(5)                                = 0
fcntl(4, F_GETFL)                       = 0 (flags O_RDONLY)
fstat(4, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb57eacc000
lseek(4, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x7fb57eacc000, 4096)            = 0
wait4(19563, [{WIFEXITED(s) && WEXITSTATUS(s) == 127}], 0, NULL) = 19563
stat("/var/lib/lxc/vagrant-local-machine-1/rootfs", {st_mode=S_IFDIR|0755, st_size=152, ...}) = 0
open("/var/lib/lxc/vagrant-local-machine-1/rootfs", O_RDONLY) = 4
ioctl(4, BTRFS_IOC_SPACE_INFO, 0x7fff1ecae980) = 0
close(4)                                = 0
stat("/var/lib/lxc/vagrant-local-machine-1/rootfs", {st_mode=S_IFDIR|0755, st_size=152, ...}) = 0
open("/var/lib/lxc/vagrant-local-machine-1", O_RDONLY) = 4
ioctl(4, BTRFS_IOC_SNAP_DESTROY, 0x7fff1ecada30) = -1 ENOTEMPTY (Directory not empty)
close(4)                                = 0
write(2, "lxc_container: ", 15lxc_container: )         = 15
write(2, "Error destroying rootfs for vagr"..., 51Error destroying rootfs for vagrant-local-machine-1) = 51
write(2, "\n", 1
)                       = 1
fcntl(3, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
close(3)                                = 0
write(2, "Destroying vagrant-local-machine"..., 42Destroying vagrant-local-machine-1 failed
) = 42
exit_group(1)                           = ?
+++ exited with 1 +++

(.venv-ubuntu)vagrant@vagrant-ubuntu-trusty-64:/ev/bzr/uci-engine/ceph$ sudo btrfs subvolume list /var/lib/lxc/vagrant-local-machine-1
ID 256 gen 7632 top level 5 path @lxc
ID 257 gen 6811 top level 256 path juju-precise-template/rootfs
ID 285 gen 7424 top level 256 path juju-trusty-template/rootfs
ID 286 gen 7625 top level 256 path vagrant-local-machine-1/rootfs
ID 288 gen 7630 top level 256 path vagrant-local-machine-3/rootfs
ID 289 gen 6443 top level 256 path vagrant-local-machine-4/rootfs
ID 311 gen 6441 top level 256 path vagrant-local-machine-4/rootfs/srv/disk/current
ID 320 gen 6444 top level 256 path vagrant-local-machine-3/rootfs/srv/disk/current
ID 336 gen 6436 top level 256 path vagrant-local-machine-1/rootfs/srv/disk/current
ID 339 gen 6432 top level 256 path vagrant-local-machine-1/rootfs/srv/disk/snap_793
ID 340 gen 6433 top level 256 path vagrant-local-machine-4/rootfs/srv/disk/snap_1349
ID 341 gen 6434 top level 256 path vagrant-local-machine-3/rootfs/srv/disk/snap_1192
ID 342 gen 6436 top level 256 path vagrant-local-machine-1/rootfs/srv/disk/snap_794
ID 343 gen 6441 top level 256 path vagrant-local-machine-4/rootfs/srv/disk/snap_1350
ID 344 gen 6444 top level 256 path vagrant-local-machine-3/rootfs/srv/disk/snap_1193
ID 345 gen 7406 top level 256 path vagrant-local-machine-5/rootfs
ID 346 gen 7410 top level 256 path vagrant-local-machine-6/rootfs
ID 347 gen 7417 top level 256 path vagrant-local-machine-7/rootfs
ID 363 gen 7402 top level 256 path vagrant-local-machine-6/rootfs/srv/disk/current
ID 380 gen 7401 top level 256 path vagrant-local-machine-7/rootfs/srv/disk/current
ID 398 gen 7399 top level 256 path vagrant-local-machine-5/rootfs/srv/disk/current
ID 627 gen 7355 top level 256 path vagrant-local-machine-5/rootfs/srv/disk/snap_6051
ID 631 gen 7396 top level 256 path vagrant-local-machine-5/rootfs/srv/disk/snap_6104
ID 632 gen 7397 top level 256 path vagrant-local-machine-7/rootfs/srv/disk/snap_6357
ID 633 gen 7398 top level 256 path vagrant-local-machine-6/rootfs/srv/disk/snap_7272
ID 634 gen 7401 top level 256 path vagrant-local-machine-7/rootfs/srv/disk/snap_7383
ID 635 gen 7402 top level 256 path vagrant-local-machine-6/rootfs/srv/disk/snap_8298

/srv/disk what I told the ceph charm to use for osd-devices.

Changed in juju-core:
status:	Expired → New

Revision history for this message

Evan (ev) wrote on 2014-07-12:

#6

Yup, that's definitely it. Deleting the subvolumes under /var/lib/lxc/vagrant-local-machine-*/rootfs/srv/disk/{current,snap_} freed up /var/lib/lxc/vagrant-local-machine-* so that lxc-destroy worked.

Revision history for this message

Evan (ev) wrote on 2014-07-14:

#7

To further clarify, this isn't a juju bug. lxc should be smart enough to delete dependent subvolumes before it deletes the rootfs subvolume. I took a stab at this over the weekend, but ran out of time. btrfs_destroy(struct bdev *orig) is what you're after.

Revision history for this message

Serge Hallyn (serge-hallyn) wrote on 2014-07-14:

#8

To be sure I understand, the issue is that inside your container you created btrfs subvolumes (by using btrfs containers or manually using btrfs subvolume create)?

Curtis Hovey (sinzui) on 2014-07-14

Changed in juju-core:
status:	New → Triaged
importance:	Medium → High
importance:	High → Medium

Revision history for this message

Evan (ev) wrote on 2014-07-17:

#9

object-storage.yaml Edit (753 bytes, text/plain)

Correct. Attached is a juju-deployer config that reproduces the issue when used with the juju local provider and /var/lib/lxc in the host backed onto btrfs. If you deploy it and run juju destroy-environment --force -y local, it should fail.

This is because Ceph sees that /srv/ceph is on btrfs and makes use of it.

ubuntu@vagrant-local-machine-1:~$ mount
/dev/sda2 on / type btrfs (rw)

So you'll end up with lots of snapshots created by Ceph inside LXC. These will be visible by running `btrfs subvolume list /var/lib/lxc` in the host:

(.venv-ubuntu)vagrant@vagrant-ubuntu-trusty-64:~$ sudo btrfs subvolume list /var/lib/lxc
ID 256 gen 9268 top level 5 path @lxc
ID 257 gen 8126 top level 256 path juju-precise-template/rootfs
ID 285 gen 8006 top level 256 path juju-trusty-template/rootfs
ID 762 gen 9208 top level 256 path vagrant-local-machine-25/rootfs
ID 763 gen 9208 top level 256 path vagrant-local-machine-26/rootfs
ID 764 gen 9208 top level 256 path vagrant-local-machine-27/rootfs
ID 925 gen 9208 top level 256 path vagrant-local-machine-27/rootfs/srv/ceph/current
ID 934 gen 9208 top level 256 path vagrant-local-machine-26/rootfs/srv/ceph/current
ID 944 gen 9208 top level 256 path vagrant-local-machine-25/rootfs/srv/ceph/current
ID 951 gen 9208 top level 256 path vagrant-local-machine-26/rootfs/srv/ceph/snap_4426
ID 954 gen 9208 top level 256 path vagrant-local-machine-26/rootfs/srv/ceph/snap_4483
ID 957 gen 9208 top level 256 path vagrant-local-machine-25/rootfs/srv/ceph/snap_4767
ID 958 gen 9208 top level 256 path vagrant-local-machine-27/rootfs/srv/ceph/snap_5753
ID 959 gen 9208 top level 256 path vagrant-local-machine-25/rootfs/srv/ceph/snap_4845
ID 960 gen 9208 top level 256 path vagrant-local-machine-27/rootfs/srv/ceph/snap_5754

Serge Hallyn (serge-hallyn) on 2014-07-18

Changed in lxc (Ubuntu):
importance:	Undecided → Medium
status:	New → Triaged

Curtis Hovey (sinzui) on 2014-08-05

tags:	added: ubuntu-engineering
Changed in juju-core:
importance:	Medium → High
milestone:	none → next-stable

Revision history for this message

Serge Hallyn (serge-hallyn) wrote on 2014-08-05:

#10

The fix for this is applied in lxc's git HEAD.

Changed in lxc (Ubuntu):
status:	Triaged → Fix Committed

Revision history for this message

Launchpad Janitor (janitor) wrote on 2014-10-02:

#11

This bug was fixed in the package lxc - 1.1.0~alpha2-0ubuntu2

---------------
lxc (1.1.0~alpha2-0ubuntu2) utopic; urgency=medium

* Cherry-pick usptream bugfix for lxc-usernic test.
-- Stephane Graber <email address hidden> Thu, 02 Oct 2014 15:01:56 -0400

Changed in lxc (Ubuntu):
status:	Fix Committed → Fix Released

Curtis Hovey (sinzui) on 2014-10-06

Changed in juju-core:
status:	Triaged → Invalid

Curtis Hovey (sinzui) on 2014-10-21

Changed in juju-core:
milestone:	next-stable → none

Affects		Status	Importance	Assigned to	Milestone
	juju-core	Invalid	High	Unassigned
	lxc (Ubuntu)	Fix Released	Medium	Unassigned

juju-core

destroy-environment fails to clear lxc containers

Bug Description

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches