Comment 51 for bug 1602192

Revision history for this message
Stéphane Graber (stgraber) wrote :

Not really. The problem is that bumping those sysctls to the value in that document will work on most normal systems but will fail dramatically on very low memory systems like some cloud instances and ARM systems.

As for having LXD know when things will fail, it simply has not idea. All the errors happen through processes inside the container that LXD has no visibility on.

You can start hundreds (possibly thousands) of alpine containers before any such problem occur, but spawning 20 containers that use systemd will trip it. Similarly, you can probably start about 10x as many Ubuntu 14.04 containers before you run the same problem as with Ubuntu 16.04 containers.

In most cases, what's receiving the allocation error back from the kernel is systemd in the container. And rather than moving on and just doing some polling when inotify doesn't work, it just plain hangs there without providing any kind of useful feedback to the user (since logging isn't even started at that point).

As I said before, there is kernel work being done now to fix the inotify part of this problem in a clean, sane way which will work for everyone. Until then, it's reasonable for Juju to bump those limits as they know exactly what kind of instance they're running on.

For the ulimits (limits.conf), we'll have to look into what's going on here because the LXD systemd unit does have those bumped, but they somehow seem to be ignored by systemd or reset at some point in time, causing some of this breakage. I suspect it'll boil down to being a systemd bug, but we need to take a close look at this.