php-fpm sometimes SIGSEGVs (signal 11) when running fpm_get_status

Hi Lars,

Thanks for taking the time to report this bug and help making Ubuntu better.

I suppose you are able to reproduce the issue constantly then?

Would you mind providing a short (reliable) reproducer for the issue I could use to verify and investigate from a fresh Ubuntu 20.04 installation?

I am setting this bug status as incomplete. Please, move it back to new once you provide the additional information.

I've tested on another 20.04.6 and I could reproduce it, here's what I did:

place this php script somewhere accessible for web-server & php-fpm:

<?php
fpm_get_status();

I used the following mods to /etc/php/7.4/fpm/pool.d/www.conf but I'm not sure it's needed

pm = static
pm.max_children = 100

then run something like apache bench or similar, for me I used something like:
ab -n 10000 -c 99 http://localhost/status.php

after a couple of seconds SIGSEGV warnings started to appear in the php-fpm log.

I think I've found what causes the SIGSEGV, see line 59 in sapi/fpm/fpm/fpm_status.c: It uses scoreboard_p after it checks it for NULL

57 scoreboard_p = fpm_scoreboard_acquire(NULL, 1);
58 if (!scoreboard_p) {
59 zlog(ZLOG_NOTICE, "[pool %s] status: scoreboard already in use.", scoreboard_p->pool); // scoreboard_p is NULL here
60 return -1;
61 }

Hope this helps.

Nice catch, Lars :)

Apparently this had been around for a while and was fixed upstream a few months ago:
https://github.com/php/php-src/commit/df259f88daaf7df5673fd78a0a1b76a1d831d0a2

Since you did all the work here so far, would you like to drive the SRU by providing debdiffs for mantic, jammy, and focal (they are all affected) and filing the SRU paperwork? I would be happy to guide you through the process in case you are not familiar with it and would sponsor (upload) the patches on your behalf.

Otherwise, I will be happy to prepare the SRUs myself.

Thank you!

Yeah sure I can try if you're willing to help. I've tried to learn about how to work with .deb packages so I know the basics but I'll probably need some guidance.

Great! All we need for SRUs should be described in https://wiki.ubuntu.com/StableReleaseUpdates.

We can start with filling the SRU bug template, as described in https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template. We want to re-write the bug description to include all those fields described there.

For the debdiff, you can fetch the patch from https://github.com/php/php-src/commit/df259f88daaf7df5673fd78a0a1b76a1d831d0a2.patch and add some dep3 headers to it (https://dep-team.pages.debian.net/deps/dep3/).

If you need help with anything, feel free to ask here or in IRC (you will find me in Libera, in #ubuntu-devel or #ubuntu-server (nickname: athos).

Thanks!

Hey Lars, I guess I missed you on IRC (sry).

To answer your question, yes, since we do have an upstream patch available, it would be better to just go ahead and backport that one! Not that it is any better or than your own patch. It is just easier to deal with changes on the same file/function/line in the future if we stick with those upstream changes!

Yeah no problem Athos, I feel like maybe you can take it from here. The official fix is not mine and I haven't even tested that one so I don't think I'm needed here. Thanks Athos.

Thanks Lars, I will proceed with the SRUs then!

This is fixed upstream in php-8.3.1, php-8.2.14, and php-8.1.27. Hence, this should be fixed in noble, but needs fixing in mantic, jammy, and focal.

Hello Lars-Göran, or anyone else affected,

Accepted php8.2 into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php8.2/8.2.10-2ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted php8.2 (8.2.10-2ubuntu2) for mantic have finished running.
The following regressions have been reported in tests triggered by the package:

php-sabre-vobject/2.1.7-6.1 (arm64, armhf, ppc64el, s390x)
php8.2/8.2.10-2ubuntu2 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/mantic/update_excuses.html#php8.2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hi guys, thanks for pushing this forward. I've found another issue related to this but I'm still investigating so not sure yet what to make of it.

Thank you, Lars!

I am about to prepare the SRUs for jammy and focal (the mantic one is already in proposed). Are you referring to the package in mantic-proposed (with the fix) or the package without the fix? i.e., is this relevant for the SRU verification?

FWIW, the regression tests were flaky or unrelated (re-trigger or migration-reference/0 got things passing again).

Reproducing the issue in mantic with the script provided in the SRU test plan:

+ echo 'Number of SIGSEGV failures:'
Number of SIGSEGV failures:
+ lxc exec php-fpm-segfault -- sh -c 'cat /var/log/php8.2-fpm.log | grep -c SIGSEGV'
5102

Which confirms the issue.

When tryong to verify the fix, I got the following result:

+ echo 'Number of SIGSEGV failures:'
Number of SIGSEGV failures:
+ lxc exec php-fpm-segfault -- sh -c 'cat /var/log/php8.2-fpm.log | grep -c SIGSEGV'
315

Further runs with and without the fix are quite consistent: I get thousands of failures for the package in the release pocket, and a few hundreds with the package in proposed.

Before marking this as a verification failure, I would like to dig a bit more to ensure this is not a different bug or a a issue with local resources.

This is the modified test plan script used to install the packages from the proposed pocket:

#!/bin/bash

set -eux

trap cleanup EXIT

UBUNTU_SERIES=${UBUNTU_SERIES:-mantic}
PHP_VERSION=${PHP_VERSION:-8.2}
TEST_CONTAINER=php-fpm-segfault
PHP_TEST_FILE=$(mktemp)
PROPOSED_REPO_FILE=$(mktemp)
PROPOSED_PREF=$(mktemp)

cleanup() {
  rm -f ${PHP_TEST_FILE}
  rm -f ${PROPOSED_REPO_FILE}
  rm -f ${PROPOSED_PREF}
  lxc delete -f ${TEST_CONTAINER}
}

cat > ${PHP_TEST_FILE} <<EOF
testing...
<br/>
<?php
phpinfo();
fpm_get_status();
EOF

cat > ${PROPOSED_REPO_FILE} <<EOF
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ ${UBUNTU_SERIES}-proposed restricted main multiverse universe
EOF

cat > ${PROPOSED_PREF} <<EOF
# Configure apt to allow selective installs of packages from proposed
Package: *
Pin: release a=${UBUNTU_SERIES}-proposed
Pin-Priority: 500
EOF

lxc launch ubuntu-daily:${UBUNTU_SERIES} ${TEST_CONTAINER}

lxc file push ${PROPOSED_REPO_FILE} ${TEST_CONTAINER}/etc/apt/sources.list.d/ubuntu-${UBUNTU_SERIES}-proposed.list
lxc file push ${PROPOSED_PREF} ${TEST_CONTAINER}/etc/apt/preferences.d/proposed-updates

lxc exec ${TEST_CONTAINER} -- apt update
lxc exec ${TEST_CONTAINER} -- apt install -y php php-fpm apache2-utils apache2 libapache2-mod-fcgid
lxc exec ${TEST_CONTAINER} -- systemctl start php${PHP_VERSION}-fpm
lxc exec ${TEST_CONTAINER} -- a2dismod php${PHP_VERSION}
lxc exec ${TEST_CONTAINER} -- a2dismod mpm_prefork
lxc exec ${TEST_CONTAINER} -- a2enconf php${PHP_VERSION}-fpm
lxc exec ${TEST_CONTAINER} -- a2enmod proxy_fcgi proxy mpm_event
lxc file push ${PHP_TEST_FILE} ${TEST_CONTAINER}/var/www/html/test.php --mode 0644
lxc exec ${TEST_CONTAINER} -- systemctl restart apache2.service
lxc exec ${TEST_CONTAINER} -- sh -c "curl -s localhost/test.php | grep -o 'FPM/FastCGI'"
lxc exec ${TEST_CONTAINER} -- ab -n 10000 -c 99 http://localhost/test.php
echo 'Number of SIGSEGV failures:'
lxc exec ${TEST_CONTAINER} -- sh -c "cat /var/log/php${PHP_VERSION}-fpm.log | grep -c SIGSEGV"

By removing the call to phpinfo(), I could get back to zero errors with the package in -proposed (and a few thousand with the package in the release pocket), meaning the observed noise was indeed unrelated to this bug/fix.

This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software: Apache/2.4.57
Server Hostname: localhost
Server Port: 80

Document Path: /test.php
Document Length: 0 bytes

Concurrency Level: 99
Time taken for tests: 0.728 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 1660000 bytes
HTML transferred: 0 bytes
Requests per second: 13729.85 [#/sec] (mean)
Time per request: 7.211 [ms] (mean)
Time per request: 0.073 [ms] (mean, across all concurrent requests)
Transfer rate: 2225.74 [Kbytes/sec] received

Connection Times (ms)
              min mean[+/-sd] median max
Connect: 0 0 0.3 0 3
Processing: 2 7 4.5 6 57
Waiting: 2 7 4.5 6 57
Total: 5 7 4.5 6 59

Percentage of the requests served within a certain time (ms)
  50% 6
  66% 7
  75% 7
  80% 7
  90% 8
  95% 9
  98% 10
  99% 12
 100% 59 (longest request)
Number of SIGSEGV failures:
0

So what are you saying, that phpinfo() is causing random segfaults? So you have a run with just phpinfo() with the php8.2 packages from release and proposed?

Hello Lars-Göran, or anyone else affected,

Accepted php8.1 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php8.1/8.1.2-1ubuntu2.16 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Lars-Göran, or anyone else affected,

Accepted php7.4 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php7.4/7.4.3-4ubuntu2.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.