Regression: Nested LXC is broken on Vivid

Bug #1436723 reported by Chris Glass
36
This bug affects 6 people
Affects Status Importance Assigned to Milestone
lxc (Ubuntu)
Fix Released
High
Unassigned
Vivid
Fix Released
Undecided
Unassigned

Bug Description

The nested LXC functionality seems to be broken on Vivid, at least with the following setup:

Vivid (Host) -> Trusty (Outer LXC) -> Trusty (Inner LXC)

What happens:

The Inner LXC start command fails when trying to start with: http://pastebin.ubuntu.com/10682639/
( 631 cgroup error? 100 cgroups with this name already running seems to be the revelant error message).

What is expected to happen:

The inner LXC start command is expected to succeed and result in a running nested container.

Steps to reproduce:

- Install vivid server daily, update.
- sudo apt-get install lxc
- sudo lxc-create -n outer -t ubuntu -- --release trusty --mirror=http://ch.archive.ubuntu.com/ubuntu -b $USER
- sudo su
- echo 'lxc.aa_profile = lxc-container-default-with-nesting' >> /var/lib/lxc/outer/config
- exit
- sudo lxc-start -n outer
- (SSH to outer)
- sudo apt-get update && sudo apt-get dist-upgrade
- sudo apt-get install lxc
- sudo lxc-create -n inner -t ubuntu -- --release trusty --mirror=http://ch.archive.ubuntu.com/ubuntu -b $USER
- sudo lxc-start -n inner

Fails with: http://pastebin.ubuntu.com/10682639/
Enabling debug logs for this action yields: http://pastebin.ubuntu.com/10682658/

Control: The exact same scenario works on trusty and utopic. Tested by spinning up server installs (from isos) in KVMs and verifying manually.

Any further log or information available on request, including KVM images demonstrating the problem.

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: lxc 1.1.0-0ubuntu1
ProcVersionSignature: Ubuntu 3.19.0-10.10-generic 3.19.2
Uname: Linux 3.19.0-10-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.16.2-0ubuntu4
Architecture: amd64
CurrentDesktop: Unity
Date: Thu Mar 26 08:43:56 2015
InstallationDate: Installed on 2015-03-12 (13 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Alpha amd64 (20150306)
SourcePackage: lxc
UpgradeStatus: No upgrade log present (probably fresh install)
defaults.conf:
 lxc.network.type = veth
 lxc.network.link = lxcbr0
 lxc.network.flags = up
 lxc.network.hwaddr = 00:16:3e:xx:xx:xx

Revision history for this message
Chris Glass (tribaal) wrote :
Chris Glass (tribaal)
tags: added: landscape
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in lxc (Ubuntu):
status: New → Confirmed
Chris Glass (tribaal)
description: updated
Revision history for this message
David Britton (dpb) wrote :

Just so it's clear, confirmed these steps are broken on an updated vivid install.

Revision history for this message
Chris Glass (tribaal) wrote :

If both the host (vivid) and the outer guest run LXC from ppa:ubuntu-lxc/daily, the tested scenario works.

sudo add-apt-repository ppa:ubuntu-lxc/daily #on both the vivid host and at leas the outer guest (trusty).

N.b. Only works if both the host and the first guest run the ppa code - seems like only the host running the newest code doesn't help.

Revision history for this message
Stéphane Graber (stgraber) wrote :

Ah, we have a new lxc and lxcfs currently stuck in vivid-proposed, I wonder if that'd include the fix you need.

Note that we're still working on fixing a bunch of regressions to allow those packages to migrate to vivid proper.

Revision history for this message
Chris Glass (tribaal) wrote :

Oh, good to hear.

Can I easily test what's in vivid-proposed, and/or somehow influence the landing? Since I have a pretty good test set up for this particular scenario I might as well help you guys :)

Revision history for this message
Stéphane Graber (stgraber) wrote :

You can temporarily turn on vivid-proposed in /etc/apt/sources.list.

lxcfs was released to vivid a couple of hours ago after the latest cgmanager landed fixing a test regressions due to systemd.

lxc is currently mostly held because of broken apparmor tests.

Revision history for this message
Chris Glass (tribaal) wrote :

So, the new lxcfs in vivid-proposed does *not* solve this particular problem.

Again, running the daily PPA code solved this, but that means using the ppa for both vivid and trusty, not just vivid.

Don't hesitate to let me know how I can test more/further versions of LXC, including git branches, if needed.

Changed in lxc (Ubuntu):
importance: Undecided → High
Revision history for this message
Chris Glass (tribaal) wrote :

This seems to be fixed in vivid with lxc 1.1.2:

tribaal@vivid-test:~$ apt-cache policy lxc
lxc:
  Installed: 1.1.2-0ubuntu1
  Candidate: 1.1.2-0ubuntu1
  Version table:
 *** 1.1.2-0ubuntu1 0
        500 http://ch.archive.ubuntu.com/ubuntu/ vivid/main amd64 Packages
        100 /var/lib/dpkg/status

tribaal@vivid-test:~$ apt-cache policy lxc-templates
lxc-templates:
  Installed: 1.1.2-0ubuntu1
  Candidate: 1.1.2-0ubuntu1
  Version table:
 *** 1.1.2-0ubuntu1 0
        500 http://ch.archive.ubuntu.com/ubuntu/ vivid/main amd64 Packages
        100 /var/lib/dpkg/status

Revision history for this message
Stéphane Graber (stgraber) wrote :

Good to hear. We did have a bunch of fixes in there which were aimed at fixing that.

Changed in lxc (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Chris Glass (tribaal) wrote :

Ok, so it seems this isn't fixed after all.

Following the exact steps reported in my original message result in a non-starting container (same error message)

Chris Glass (tribaal)
Changed in lxc (Ubuntu):
status: Fix Released → Confirmed
Revision history for this message
Chris Glass (tribaal) wrote :

Adding ppa:ubuntu-lxc/daily seems to solve the problem.

Revision history for this message
Vincent Ladeuil (vila) wrote :

I run into the same issue on a vivid host while trying to start a trusty nested container from inside a trusty container.

Revision history for this message
Vincent Ladeuil (vila) wrote :

@Chris: Where do you add the ubuntu-lxc/daily ppa ? On the host or on the outer container ?

Revision history for this message
Chris Glass (tribaal) wrote :

On the host.

Sorry for taking so long to answer, launchpad seems to have eaten my email answer...

Revision history for this message
Chris Glass (tribaal) wrote :

Correction, on the host and on outer (just tried).

Revision history for this message
Vincent Ladeuil (vila) wrote :

@Chris: Thanks !

I had to implement a different scheme not using nested containers but still requiring lxc-1.1.2.

I'll give nested containers another shot asap.

Changed in lxc (Ubuntu):
status: Confirmed → Fix Released
Changed in lxc (Ubuntu Vivid):
status: New → In Progress
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Chris, or anyone else affected,

Accepted lxc into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in lxc (Ubuntu Vivid):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Chris J Arges (arges) wrote :

Hello Chris, or anyone else affected,

Accepted lxc into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 1.1.5-0ubuntu0.15.04.3

---------------
lxc (1.1.5-0ubuntu0.15.04.3) vivid-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix preserve_ns to work on < 3.8 kernels. (LP: #1516971)
    - Fix process title rewrite to not mangle the environment. (LP: #1517107)

lxc (1.1.5-0ubuntu0.15.04.2) vivid-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix ubuntu-cloud template to detect compression algorithm instead
      of hardcoding xz. Also update list of supported releases and use trusty
      as the fallback release. (LP: #1515463)
  * Update lxc-tests description to make it clear that this package is
    meant to be used by developers and by automated testing.

lxc (1.1.5-0ubuntu0.15.04.1) vivid-proposed; urgency=medium

  * New upstream bugfix release (MRE) (1.1.5)
    (LP: #1497420, LP: #1436723, LP: #1441068, LP: #1504496,
     LP: #1466458, LP: #1510619)
  * Drop proxy detection from the autopkgtest exercise script.

 -- Stéphane Graber <email address hidden> Wed, 18 Nov 2015 13:41:23 -0500

Changed in lxc (Ubuntu Vivid):
status: Fix Committed → Fix Released
Revision history for this message
Stéphane Graber (stgraber) wrote : Update Released

The verification of the Stable Release Update for lxc has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.