apachectl configtest segfault with request-tracker4 + mod_perl

I'm not sure why I missed this when we were looking into it on IRC, but this is apparently something involving the openssl and nss crypto libraries / frameworks. I'm not enough of a developer to understand what exactly seems to be the problem, though, but knowing that apache httpd supports HTTPS with not just openssl I'd check which of these modules are enabled, and maybe make sure only one of them is used temporarily while you check whether you can still reproduce. I.e. look for 'ssl' (openssl) and something 'nss'-like in /etc/apache2/modes-enabled/ and maybe try disabling or re-enabling one of those.

I also think you should still try to
  sudo apport-collect 1854395
after reproducing this segfault.

Hi Tom.

The ssl apache module is enabled
There's nothing about nss in mods-enabled/ (but kernel message mentions libnss_files-2.23.so

When the ssl apache module is disabled, apachectl configtest works.
When the ssl apache module is enabled, apachectl configtest fails with segmentation fault.

About apport-collect.. I installed it and tried to run "apport-collect 1854395", it opens links but it will not let me login.

Thank you and best regards
Roberto

Hi.

I just:
1) apt install libapache2-mod-nss
2) a2enmod nss
3) systemctl restart apache2.service

but nothing has changed.. so I'm now going to disable and deinstall libapache2-mod-nss

Hmm, sorry for putting you on the wrong path there.

We'll now need to wait for a developer / package maintainer to have a look at this.

I just found the config line that makes it crash:

Include /etc/request-tracker4/apache2-modperl2.conf
I still have to look into it, but for now I commented it out.

Roberto

Hello Roberto,

The segfault you observed is very likely to be a bug, however I tried to reproduce it by installing/enabling request-tracker4, rt4-apache2, libapache2-mod-nss and few other things on Xenial but without success. Is there anything relevant I could try to trigger the crash?

If I understand correctly you started experiencing the problem after an upgrade, so you don't have the steps to reproduce handry from a fresh setup. We could be lucky with a strace: could you please uncomment the Include (and so get apache2ctl crash again), then run

  sudo strace apache2ctl configtest

and attach the output to this report? Before making it public better check it doesn't contain private information.

As we're waiting for additional information I'm setting the report status to Incomplete, please set the status back to New after replying. Thanks!

Hello Paride.

I don't know what else may be relevant to trigger the crash.
libapache2-mod-nss is not relevant at all.. I just installed it in an attemt to fix the problem but it didn't help.

I'm willing to help debug this, but I need guidance. So please don't hesitate to ask.

I will now copy/paste here the output of strace as you suggested.

#strace apache2ctl configtest > /root/out 2> /root/err

# cat /root/out
Action 'configtest' failed.
The Apache error log may have more information.

# cat /root/err
execve("/usr/sbin/apache2ctl", ["apache2ctl", "configtest"], [/* 22 vars */]) = 0
brk(NULL) = 0x55fa99d60000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=37741, ...}) = 0
mmap(NULL, 37741, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe1fd4cd000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1868984, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1fd4cc000
mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe1fcee8000
mprotect(0x7fe1fd0a8000, 2097152, PROT_NONE) = 0
mmap(0x7fe1fd2a8000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7fe1fd2a8000
mmap(0x7fe1fd2ae000, 14752, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe1fd2ae000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1fd4cb000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe1fd4ca000
arch_prctl(ARCH_SET_FS, 0x7fe1fd4cb700) = 0
mprotect(0x7fe1fd2a8000, 16384, PROT_READ) = 0
mprotect(0x55fa98cbc000, 8192, PROT_READ) = 0
mprotect(0x7fe1fd4d7000, 4096, PROT_READ) = 0
munmap(0x7fe1fd4cd000, 37741) = 0
getuid() = 0
getgid() = 0
getpid() = 25420
rt_sigaction(SIGCHLD, {0x55fa98ab0540, ~[RTMIN RT_1], SA_RESTORER, 0x7fe1fcf1d4b0}, NULL, 8) = 0
geteuid() = 0
brk(NULL) = 0x55fa99d60000
brk(0x55fa99d81000) = 0x55fa99d81000
getppid() = 25418
stat("/etc/apache2/sites-enabled", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/usr/sbin/apache2ctl", O_RDONLY) = 3
fcntl(3, F_DUPFD, 10) = 10
close(3) = 0
fcntl(10, F_SETFD, FD_CLOEXEC) = 0
geteuid() = 0
getegid() = 0
rt_sigaction(SIGINT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {0x55fa98ab0540, ~[RTMIN RT_1], SA_RESTORER, 0x7fe1fcf1d4b0...

I seem to have the exact same issue, also with RT4 on Ubuntu 16.04.6 LTS. The strace looks pretty much exactly the same, and the other details in the case is exactly the same, with one difference; It crashes even if I comment out the 'Include /etc/request-tracker4/apache2-modperl2.conf' line.

Note: when the line is commented out, RT stops working. I guess it was there for a reason.

Hi Roberto,

While I think you ran into an actual bug, without steps to reproduce it is very difficult for a developer to begin working on it. Do you think you could try to reproduce the segfault within a container and identify the minimal (or close to minimal) steps needed to reproduce the problem?

If you have the 'lxd' snap installed running a container is very easy, just:

$ lxc launch ubuntu:xenial xenial-test-contaioner
$ lxc exec xenial-test-container bash

and you'll get a root shell within a clean Xenial container. Thanks!

Hi Paride.
I will try the steps you propose this evening and let you know.

Kjell, do you have other web apps that use mod-perl? If so, try to disable them and let us know.

Thank you and best regards.
Roberto

Hi Paride.

that lets me with a container without any IPv4 address.. I cannot
install any packages. Sorry.. I never worked with containers.

Roberto

Il 10.12.2019 16:12, Paride Legovini ha scritto:
> Hi Roberto,
>
> While I think you ran into an actual bug, without steps to reproduce it
> is very difficult for a developer to begin working on it. Do you think
> you could try to reproduce the segfault within a container and identify
> the minimal (or close to minimal) steps needed to reproduce the problem?
>
> If you have the 'lxd' snap installed running a container is very easy,
> just:
>
> $ lxc launch ubuntu:xenial xenial-test-contaioner
> $ lxc exec xenial-test-container bash
>
> and you'll get a root shell within a clean Xenial container. Thanks!
>

--
Roberto Nunnari

Roberto: no other web apps on the box, only RT4

in addition to the mentioned include-line, I did have these lines in the 000-default and default-ssl.conf:
<Perl>
    use Plack::Handler::Apache2;
    Plack::Handler::Apache2->preload("/usr/share/request-tracker4/libexec/rt-server");
</Perl>

When commenting those out in addition to the include-line, I no longer get the error.

Also, FWIW, I tried setting up lxc test-container, have not yet been able to reproduce

Perhaps of relevance: I have done release-upgrade of the box this runs on, from ubuntu 14.04. This was done quite some time ago.

in case it helps anyone, I found this:
https://forum.bestpractical.com/t/apache-core-dump-when-rt4-is-activated/33695/3
where its suggested not to use mod_perl. I changed to an alternate method and it appears to run smoothly. It might be related to different versions of openssl compiled into apache vs perl. If your setup is like mine where it has survived release-upgrade that might be part of the picture as well.

Kjell, indeed the configuration you're showing in comment #13 is indeed a mod_perl config, so if the issue goes away with that removed, it fits Roberto's findings.

Roberto and Kjell,

Thanks for your continued investigation into this issue. It sounds like you've found an effective workaround by removing mod_per and using fastcgi or similar. It is interesting to hear that upstream considers mod_perl to not be a supportable configuration. For the Ubuntu Server team, I also suspect mod_perl configurations are not going to be priority issues for distro-level support. mod_perl is tricky and far too easy for non-experts to misconfigure.

However, before closing this as a wontfix bug, one question. How did the request-tracker4 package get configured to use mod_perl in the first place? If the debian/ubuntu packaging set it up that way, then this may be something worth us changing, whereas on the other hand if that was something you've done manually yourself (e.g. following an installation tutorial), then perhaps not.

Bryce,

In my case when I re-set it up in the container I had to do it manually (method wasn't chosen by the package).

I would assume mod_perl often gets chosen because of https://rt-wiki.bestpractical.com/wiki/ManualApacheConfig which has examples for mod_perl but not for the others, as well as lack of warnings there.

As for mod_perl the default config was not touched to my knowledge, FWIW (but yes, it is tricky)

I suspect that having this set up and doing a release-upgrade (or perhaps two) might not have helped. To my knowledge I configured the container as close to our production system as I could, and I was not able to replicate the issue. I am sure I missed something, I just don't know what and I don't have cycles to dig deeper.

Since there are effective workarounds available I would be OK with this getting closed as wontfix.

Even when closing this as "won't Fix" as agreed I really want to thank you @Kjell for your participation and detailed reports. Good bugs are rare and this is one.

Ok for me to close it as "won't fix". Just use mod-fastcgi instead of mod-perl.
Christian, no need to thank me for reporting this bug, giving all the detailed reports, and hinting mod-perl as the triggering cause of this segmentation fault.
Best regards.
Roberto