can't delete docker containers in lxd: Driver btrfs failed to remove root filesystem

Docker is supposed to always be using aufs inside LXD containers, the output above suggests it's not the case.

btrfs doesn't allow for unprivileged btrfs subvolume delete by default, only subvolume creation (yes, that's a bit odd). So the failure you're getting is absolutely normal except for the fact that I was told Docker would always do aufs inside unprivileged LXD containers.

As a workaround to keep having docker do btrfs inside LXD, you may want to change the host btrfs mountpoint to have user_subvol_rm_allowed as a mount option, then restart the LXD container and it should be able to both create and delete subvolumes.

Note that a bunch more btrfs functions will still fail in the container, so it'll likely only make things slightly better for you.

Thank you for this clarification.

I'm not sure how to enable the aufs backend. Reading in the docker documentation at https://docs.docker.com/engine/userguide/storagedriver/selectadriver/, I understand with lxc running on btrfs, there is no other option than using the btrfs option.

http://serverfault.com/questions/615588/proper-way-of-handling-lxc-containers-on-btrfs/676504#676504

Status changed to 'Confirmed' because the bug affects multiple users.

Not sure if Docker, LXD, or btrfs. "user_subvol_rm_allowed" is enabled, issue only started appearing recently, unsure if due to Docker upgrade or LXD upgrade. Observed this error earlier but "user_subvol_rm_allowed" resolved it. However, with it enabled it still occurs now.

me@host: lxd --version
2.4.1

root@docker:~# docker info
Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 1
Server Version: 1.12.1
Storage Driver: btrfs
 Build Version: Btrfs v4.7
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 4.4.0-43-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 4 GiB
Name: docker
ID: FOAC:LFCO:6B2U:HWVU:NGEC:GZAS:ZKCI:EBIU:JDBP:EGB4:2NMN:4KXI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

root@docker:~# docker --version
Docker version 1.12.1, build 23cf638

root@docker:~# mount | grep btrfs
/dev/md2 on / type btrfs (rw,relatime,space_cache,user_subvol_rm_allowed,subvolid=49300,subvol=/var/lib/lxd/containers/docker/rootfs)
/dev/md2 on /dev/lxd type btrfs (rw,relatime,space_cache,user_subvol_rm_allowed,subvolid=5,subvol=/var/lib/lxd/devlxd)
/dev/md2 on /dev/.lxd-mounts type btrfs (rw,relatime,space_cache,user_subvol_rm_allowed,subvolid=5,subvol=/var/lib/lxd/shmounts/docker)
/dev/md2 on /var/lib/docker/btrfs type btrfs (rw,relatime,space_cache,user_subvol_rm_allowed,subvolid=49300,subvol=/var/lib/lxd/containers/docker/rootfs/var/lib/docker/btrfs)

And finally, the error:

docker rm 681fbe3f6870
Error response from daemon: Driver btrfs failed to remove root filesystem 0c214cb543ccd067e5208c8d5374b8345ec534c54b488719f17adb5f6cd44821: Failed to rescan btrfs quota for &{}: operation not permitted

I also observe sporadic errors with:

Error response from daemon: stat /var/lib/docker/btrfs/subvolumes/a28d807f83e6131a6e43e062a73e7cbff496746e2dfceecc39a27483cabcd0d4: no such file or directory