systemd does not notice when services created via systemd-sysv-generator fail

Bug #1488962 reported by Dr. Jens Harbott
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
apache2 (Ubuntu)
Fix Released
Medium
Kick In

Bug Description

The systemd service generated from /etc/init.d/apache2 via systemd-sysv-generator contains the line

RemainAfterExit=yes

causing systemd to ignore crashes of the service. In order to reproduce this, add a non-existing address to /etc/apache2/ports.conf, which will cause the apache2 service to fail.

However, "systemctl status apache2" will still show "active(exited)", e.g. will not recognize the service as failed. In particular, "systemctl is-failed apache2" will still return "active" instead of "failed".

Expected result: "systemctl is-failed apache2" should return "failed", so that monitoring script can detect the error.

The solution would probably be to include a proper native systemd service definition instead of using the converted sysv service.

$ apt-cache policy apache2
apache2:
  Installed: 2.4.10-9ubuntu1.1
  Candidate: 2.4.10-9ubuntu1.1
  Version table:
 *** 2.4.10-9ubuntu1.1 0
        500 http://eu.archive.ubuntu.com/ubuntu/ vivid-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ vivid-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.10-9ubuntu1 0
        500 http://eu.archive.ubuntu.com/ubuntu/ vivid/main amd64 Packages

Robie Basak (racb)
tags: added: systemd-boot
Robie Basak (racb)
Changed in apache2 (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apache2 (Ubuntu):
status: New → Confirmed
Revision history for this message
cd311 (cd311) wrote :

Hi

apparently there is a patch available which is included in Suse but not yet accepted by systemd.
See https://bugzilla.novell.com/show_bug.cgi?id=721426

Jon Grimm (jgrimm)
Changed in apache2 (Ubuntu):
assignee: nobody → Kick In (kick-d)
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

My first try to verify the current state was with the ports.conf trick, but that failed me (as it worked).

So I looked for other ways to reproduce.
The linked suse bug report used kill to recreate - and this worked for me.

Ok here we have our simplified test (not even needs to change the conf now) and the verification that it is still an issue in Xenial.

reset the conf , restart apache
killall -9 apache2
systemctl status apache2
● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2; bad; vendor preset: enabled)
   Active: active (exited) since Wed 2016-04-06 16:05:43 UTC; 48s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1109 ExecStop=/etc/init.d/apache2 stop (code=exited, status=0/SUCCESS)
  Process: 1152 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)

Apr 06 16:05:43 xenial-tgt-tests systemd[1]: Starting LSB: Apache2 web server...
Apr 06 16:05:43 xenial-tgt-tests apache2[1152]: * Starting web server apache2
Apr 06 16:05:43 xenial-tgt-tests apache2[1152]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127
Apr 06 16:05:43 xenial-tgt-tests apache2[1152]: *
Apr 06 16:05:43 xenial-tgt-tests systemd[1]: Started LSB: Apache2 web server.

That said - the behavior is not limited to apache - e.g. also nscd.
But on the other hand others (probably with real own service files) work - tested on openssh-server.
The current assumption is - as also mentioned in the initial report that it applies to all that come via systemd-sysv-generator.

summary: - systemd does not notice when apache2 service fails
+ systemd does not notice when services created via systemd-sysv-generator
+ fail
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

This could be another "this is how it is supposed in systemd, but there is a totally different way ...".
Adding Martin Pitt to the subscribers for his experience on that.

Revision history for this message
Martin Pitt (pitti) wrote :

SysV init scripts don't have enough defined machine parseable semantics to tell what they do. Some just run a few commands (the equivalent of systemd's "Type=oneshot", others fork off a daemon (the equivalent of "Type=forking"). Thus for SysV generated units, systemd defaults to RemainAfterExit=yes as that's the safer default -- not doing so would assume that the SysV script is idempotent. So I'm afraid this default won't/can't be changed in a sensible way.

If you want to improve the behaviour particularly for Apache, then indeed adding native service units would be best. This can then use modern features such as socket activation, which vastly simplifies inter-service dependencies, and avoids running apache unnecessarily. E. g. Fedora does that (http://pkgs.fedoraproject.org/cgit/rpms/httpd.git/tree/ has the units)

If that's too much effort for Xenial, then a simpler way would be to just ship a drop-in `/lib/systemd/system/apache2.service.d/type.conf with "[Service]\nType=forking", so that you just override this one property from the generated unit.

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

I think doing socket based activation properly is non-trivial, as there are people wanting apache to run only on non-standard ports, e.g. when deploying OpenStack services. So having a default setup that causes systemd to listen on port 80, without checking whether the apache config really does want that, would be pretty bad IMHO. Same thing for mod_ssl and port 443. And it looks to me like that is what Fedora does, so please don't just copy that behaviour.

So I would vote for doing the Type=forking solution in Xenial now in order to get the original issue fixed in time for release and develop some sensible native units later.

Revision history for this message
Kick In (kick-d) wrote :
Changed in apache2 (Ubuntu):
status: Confirmed → In Progress
Revision history for this message
Martin Pitt (pitti) wrote :

Calling the drop-in apache2.service.d/apache2-systemd.conf sounds a bit redundant, I think a name like forking.conf would be a bit more expressive. But this is just a nitpick, looks good to me in general.

Revision history for this message
Kick In (kick-d) wrote : Re: [Bug 1488962] Re: systemd does not notice when services created via systemd-sysv-generator fail

Yes, but I wanted to rename it with dh-exec, but it requires debhelper
 >9, and I thought creating a subdir just for that file would have been
overkill.

But I can change the name.

Le 08/04/2016 12:18, Martin Pitt a écrit :
> Calling the drop-in apache2.service.d/apache2-systemd.conf sounds a bit
> redundant, I think a name like forking.conf would be a bit more
> expressive. But this is just a nitpick, looks good to me in general.
>

Revision history for this message
Kick In (kick-d) wrote :

(debhelper >= 9)

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Louis Bouchard (louis)
tags: added: sts-sponsor
Jon Grimm (jgrimm)
Changed in apache2 (Ubuntu):
milestone: none → ubuntu-16.04
Revision history for this message
Kick In (kick-d) wrote :

For reference, a systemd discussion for apache2 in a debian bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798430

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apache2 - 2.4.18-2ubuntu2

---------------
apache2 (2.4.18-2ubuntu2) xenial; urgency=medium

  * Correct systemd-sysv-generator behavior by customizing some parameters (LP: #1488962)
    - d/apache2-systemd.conf: add a drop-in file to specify some parameters for the systemd
      unit (type=Forking and RemainsAfterExit=no), this allow a correct state synchronisation
      between systemctl status and actual state of apache2 daemon.
    - d/apache2.install: place the apache2-systemd.conf file in the correct location.

 -- Pierre-André MOREY <email address hidden> Fri, 08 Apr 2016 11:48:00 +0200

Changed in apache2 (Ubuntu):
status: In Progress → Fix Released
Louis Bouchard (louis)
tags: removed: sts-sponsor
Revision history for this message
Luis Michael Ibarra (clvx) wrote :

This problem is still going on:

$ sudo apt-cache policy apache2
apache2:
  Installed: 2.4.18-2ubuntu3
  Candidate: 2.4.18-2ubuntu3
  Version table:
 *** 2.4.18-2ubuntu3 500
        500 http://azure.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status

$ sudo systemctl status apache2.service
● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2; bad; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: active (exited) since Sun 2016-06-19 00:01:35 UTC; 16min ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 0
   Memory: 0B
      CPU: 0

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.