MAAS

[SRU] avahi fails in containers

Bug #1251257 reported by Daniel Westervelt
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
High
Unassigned
avahi (Ubuntu)
Fix Released
Critical
Unassigned
Precise
Fix Released
Critical
Unassigned
Saucy
Fix Released
Critical
Unassigned
Trusty
Fix Released
Critical
Unassigned

Bug Description

installed a brand new maas server on suacy into an lxc container from archive and http://<ip>/MAAS is not accessible although http://<ip> is accessible

http://<ip>/MAAS is getting logged to /var/log/apache2/access.log though so request is making it through

<roaksoax> danwest: can you please pastebin apache2's error and access.log
<danwest> http://paste.ubuntu.com/6405411/
<danwest> http://paste.ubuntu.com/6405412/

<roaksoax> danwest: I know what it is
<roaksoax> danwest: dbus/avahi
<roaksoax> danwest: try to restart whatever avahi service there is
<andreas> "dbus" reminds me of https://bugs.launchpad.net/juju-core/+bug/1248283
<andreas> but that was about the units, not maas itself
<danwest> https://pastebin.canonical.com/100290/
<danwest> same problem
<roaksoax> danwest: yeah the avahi daemon is failing to start, causing maas to fail
<roaksoax> danwest: maybe restart whatever dbus serviice it is, and then the avahi-daemon
<matsubara> danwest, restart dbus and avahi-daemon, see https://bugs.launchpad.net/maas/+bug/1221059

<roaksoax> danwest: did avahi-daemon restart corrrectly?
<danwest> nope
<roaksoax> danwest: i guess then an issue with dbus is preventing avahi from working... hence maas failing
<danwest> roaksoax: should not matter but the only thing that is a little unique is that this is in a saucy container

<roaksoax> danwest: ah so then thats the issue...
<danwest> what, the container?
<roaksoax> yeah
<danwest> how so?
<roaksoax> avahi might have issues running in a container

<danwest> hallyn: roaksoax: what should I file that lxc/avahi /maas bug that I hit this morning against?
<hallyn> danwest: i think maas should work around it by unsetting rlimit-nproc
<hallyn> (and/or by running on trusty in a private user ns

<hallyn> smoser: fwiw the problem is that avahi sets its nproc rlimit to exaclty 3, but in a container it's using a uid that is in use on the host - so it exceeds 3 tasks
<hallyn> (i.e. it's reusing uid which is ntp on the host, and ntp is running; or just another avahi)
<smoser> ok...
<smoser> so that doesn't seem like maas's problem to me
<smoser> nor juju's really.
<hallyn> smoser: it is. it needs to pick a unique uid, or configure avahi to ignore the rlimit
<smoser> maas isn't running avahi
<smoser> is it ?
<hallyn> i duno what's actually running it :) it's *for* maas, but it probably is juju
<smoser> what if there was a bug in php, and a user used maas to deploy php.
<smoser> should we fix that in maas ?
<hallyn> you're talking about a bug. i'm talking about a resource conflict
<hallyn> having avahi alwasy run without nprocs, for protection, would be wrong for this.

fix is still up for debate on this one...

[Impact]
Avahi sets the rlimit_nproc to 3, causing avahi to fail running in containers. This This option should not be set in containers at all. This causes avahi-daemon to fail, hence all the applications that use avahi will also fail. In this particular case, MAAS fails because of this.

[Test Case]
1. Install a container.
2. Install MAAS
3. Check apache2 log for errors, such as those in [1].

[Regression Potential]
Minimal. This has been tested and works as expected.

[1]: http://paste.ubuntu.com/6405412/

See original description

Related branches

lp:~allenap/maas/death-to-avahi
Julian Edwards (community): Approve
Diff: 535 lines (+7/-373)
15 files modified
HACKING.txt (+3/-3)
buildout.cfg (+0/-4)
contrib/maas-http.conf (+1/-2)
docs/nodes.rst (+0/-9)
docs/troubleshooting.rst (+2/-8)
required-packages/base (+0/-3)
required-packages/dev (+0/-1)
setup.py (+0/-2)
src/maasserver/maasavahi.py (+0/-56)
src/maasserver/start_up.py (+0/-4)
src/maasserver/tests/test_maasavahi.py (+0/-119)
src/maasserver/tests/test_start_up.py (+0/-8)
src/maasserver/tests/test_zeroconfservice.py (+0/-84)
src/maasserver/views/nodes.py (+1/-1)
src/maasserver/zeroconfservice.py (+0/-69)
lp:~allenap/maas/death-to-avahi-packaging
Julian Edwards (community): Approve
Diff: 39 lines (+4/-4)
2 files modified
debian/changelog (+3/-0)
debian/control (+1/-4)
lp:ubuntu/trusty-proposed/avahi
lp:ubuntu/saucy-proposed/avahi
lp:ubuntu/precise-proposed/avahi
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

If you can run without avahi (as someone suggested) that would be ideal.

Otherwise, assuming juju is being used to set up avahi (even indirectly through dependencies), juju should sed -i '/rlimit-nproc/d' /etc/avahi/avahi-daemon.conf.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

For a bit of unhelpful history, years ago there was a mostly-unhelpful user namespace implementation which did solve this, by giving separate accounting for userids in each namespace. That was removed in favor of the new full implementation which will be available in trusty, and which can be used to fix this issue.

Revision history for this message
Gavin Panella (allenap) wrote :

MAAS should not keel over or get blocked if Avahi is not responsive or otherwise broken. I thought some work was done on this recently, but perhaps it didn't make it into saucy's package.

Changed in maas:
status: New → Triaged
Revision history for this message
Daniel Westervelt (danwest) wrote :

assume juju is not setting up avahi, in my case I just created a new container and did an apt-get install on MAAS.

if user NS can fix it in trusty that is great, lets document how and track it through.

for saucy, we need to figure out a fix or seemless workaround as we need to be able to use/test/demo this functionality right now.

Revision history for this message
Gavin Panella (allenap) wrote :

I seem to be mistaken about that. However, we would like to drop the zeroconf/avahi stuff in MAAS. It's there only to support the 'Install with MAAS' option from the CD installer, which we're hoping can be dropped too. There's no evidence that it's being used, and removing it would let us get rid of several moving parts that have caused us issues in the past, and continue to do so (as evidenced by this bug).

Revision history for this message
Gavin Panella (allenap) wrote :

I've attached a couple of optimistic branches here, so we can pull the trigger at the first hint.

Revision history for this message
Daniel Westervelt (danwest) wrote :

It is on me to find out if we can drop it. Either way we still need a fix for saucy.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

http://people.canonical.com/~serge/avahi-nproc.debdiff has a debdiff which solves this by not setting nproc rlimit in a container.

(The debdiff is for saucy, but should work for trusty too).

Andres Rodriguez (andreserl)
Changed in avahi (Ubuntu):
importance: Undecided → Critical
Changed in avahi (Ubuntu Saucy):
importance: Undecided → Critical
Changed in maas:
status: Triaged → Invalid
status: Invalid → Confirmed
status: Confirmed → Triaged
Andres Rodriguez (andreserl)
description: updated
Andres Rodriguez (andreserl)
summary: - MAAS install in lxc fails due to avahi issues
+ [SRU] avahi fails in containers
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package avahi - 0.6.31-2ubuntu5

---------------
avahi (0.6.31-2ubuntu5) trusty; urgency=low

  * d/p/skip-nproc-in-container.patch: Detect whether we are running in a
    container, and do not set rlimit_nproc if so (LP: #1251257)
 -- Serge Hallyn <email address hidden> Mon, 18 Nov 2013 20:44:14 +0000

Changed in avahi (Ubuntu Trusty):
status: New → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Daniel, or anyone else affected,

Accepted avahi into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/avahi/0.6.31-2ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in avahi (Ubuntu Saucy):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
Andres Rodriguez (andreserl) wrote :

I've tested this and works as expected. Marking verification-done.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Stéphane Graber (stgraber) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package avahi - 0.6.31-2ubuntu4.1

---------------
avahi (0.6.31-2ubuntu4.1) saucy-proposed; urgency=low

  * d/p/skip-nproc-in-container.patch: Detect whether we are running in a
    container, and do not set rlimit_nproc if so (LP: #1251257)
 -- Serge Hallyn <email address hidden> Mon, 18 Nov 2013 20:44:14 +0000

Changed in avahi (Ubuntu Saucy):
status: Fix Committed → Fix Released
Andres Rodriguez (andreserl)
Changed in avahi (Ubuntu Precise):
importance: Undecided → Critical
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Daniel, or anyone else affected,

Accepted avahi into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/avahi/0.6.30-5ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in avahi (Ubuntu Precise):
status: New → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Andres Rodriguez (andreserl) wrote :

I've tested this and works as expected!

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package avahi - 0.6.30-5ubuntu2.1

---------------
avahi (0.6.30-5ubuntu2.1) precise-proposed; urgency=low

  [ Serge Hallyn ]
  * d/p/skip-nproc-in-container.patch: Detect whether we are running in a
    container, and do not set rlimit_nproc if so (LP: #1251257)
 -- Andres Rodriguez <email address hidden> Wed, 11 Dec 2013 12:19:38 -0500

Changed in avahi (Ubuntu Precise):
status: Fix Committed → Fix Released
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

MAAS trunk no longer includes Avahi, so closing this bug there.

Changed in maas:
status: Triaged → Fix Released
Andres Rodriguez (andreserl)
no longer affects: maas/1.4
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.