mod_php gets disabled during do-release-upgrade

Thanks for the reproduction steps, marking as triaged.

With a fix landing for LP: #1850933, I thought I'd test upgrading from Bionic to Focal. Unfortunately is doesn't work:

root@ba:~# do-release-upgrade -d
...
libapache2-mod-php7.4: php7.2 module already enabled, not enabling PHP 7.4
Setting up apache2 (2.4.41-4ubuntu2) ...
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xe" for details.
invoke-rc.d: initscript apache2, action "restart" failed.
● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2020-04-13 17:58:23 UTC; 5ms ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 29604 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)

Apr 13 17:58:23 ba systemd[1]: Starting The Apache HTTP Server...
Apr 13 17:58:23 ba apachectl[29607]: apache2: Syntax error on line 146 of /etc/apache2/apache2.conf: Syntax error on line 3 of /etc/apache2/mods-enabled/php7.2.load: Cannot load /usr/lib/apache2/modules/libphp7.2.so into server: /usr/lib/apache2/modules/libphp7.2.so: cannot open shared object file: No such file or directory
Apr 13 17:58:23 ba apachectl[29604]: Action 'start' failed.
Apr 13 17:58:23 ba apachectl[29604]: The Apache error log may have more information.
Apr 13 17:58:23 ba systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Apr 13 17:58:23 ba systemd[1]: apache2.service: Failed with result 'exit-code'.
Apr 13 17:58:23 ba systemd[1]: Failed to start The Apache HTTP Server.
...

The problem persists after the system is rebooted post-upgrade. It looks like this on the Bionic upgraded to Focal system:

root@ba:~# dpkg -l| grep php
ii libapache2-mod-php 2:7.4+75 all server-side, HTML-embedded scripting language (Apache 2 module) (default)
rc libapache2-mod-php7.2 7.2.24-0ubuntu0.18.04.3 amd64 server-side, HTML-embedded scripting language (Apache 2 module)
ii libapache2-mod-php7.4 7.4.3-4ubuntu1 amd64 server-side, HTML-embedded scripting language (Apache 2 module)
ii php-common 2:75 all Common files for PHP packages
ii php7.4-cli 7.4.3-4ubuntu1 amd64 command-line interpreter for the PHP scripting language
ii php7.4-common 7.4.3-4ubuntu1 amd64 documentation, examples and common module for PHP
ii php7.4-json 7.4.3-4ubuntu1 amd64 JSON module for PHP
ii php7.4-opcache 7.4.3-4ubuntu1 amd64 Zend OpCache module for PHP
ii php7.4-readline 7.4.3-4ubuntu1 amd64 readline module for PHP

root@ba:~# ll /etc/apache2/mods-enabled/ | grep php
lrwxrwxrwx 1 root root 29 Apr 13 17:55 php7.2.conf -> ../mods-available/php7.2.conf
lrwxrwxrwx 1 root root 29 Apr 13 17:55 php7.2.load -> ../mods-available/php7.2.load

root@ba:~# apache2ctl -S
apache2: Syntax error on l...

This looks like a genuine bug for apache upgrade path. Triaged.

The fix for 1850933 is only handling 7.2->7.4 and 7.3->7.4 upgrades. Presumably a similar fix could be done for other upgrade paths to 7.4, but an SRU would probably be needed for 7.0->7.2 upgrades.

However, the more pressing issue would be why the 7.2->7.4 upgrade done in comment #2 did not work as intended by the fix.

I'm reproducing a failure using clean lxc:

Preparing to unpack .../25-apache2-bin_2.4.41-4ubuntu2_amd64.deb ...
Unpacking apache2-bin (2.4.41-4ubuntu2) over (2.4.29-1ubuntu4.13) ...
Preparing to unpack .../26-libapache2-mod-php_2%3a7.4+75_all.deb ...
Unpacking libapache2-mod-php (2:7.4+75) over (1:7.2+60ubuntu1) ...
(Reading database ... 32550 files and directories currently installed.)
Removing libapache2-mod-php7.2 (7.2.24-0ubuntu0.18.04.3) ...
Package apache2 is not configured yet. Will defer actions by package libapache2-mod-php7.2.
Selecting previously unselected package libapache2-mod-php7.4.
(Reading database ... 32543 files and directories currently installed.)
Preparing to unpack .../libapache2-mod-php7.4_7.4.3-4ubuntu1_amd64.deb ...
Unpacking libapache2-mod-php7.4 (7.4.3-4ubuntu1) ...

...
Setting up libapache2-mod-php7.4 (7.4.3-4ubuntu1) ...
Package apache2 is not configured yet. Will defer actions by package libapache2-mod-php7.4.

Creating config file /etc/php/7.4/apache2/php.ini with new version
libapache2-mod-php7.4: php7.2 module already enabled, not enabling PHP 7.4
Setting up packagekit-tools (1.1.13-2ubuntu1) ...
Setting up apache2 (2.4.41-4ubuntu2) ...
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xe" for details.
invoke-rc.d: initscript apache2, action "restart" failed.
● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2020-04-14 18:16:42 UTC; 4ms ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 56492 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)

Apr 14 18:16:42 php-fix-lp1865218-mod-php-upgrade systemd[1]: Starting The Apache HTTP Server...
Apr 14 18:16:42 php-fix-lp1865218-mod-php-upgrade apachectl[56496]: apache2: Syntax error on line 146 of /etc/apache2
/apache2.conf: Syntax error on line 3 of /etc/apache2/mods-enabled/php7.2.load: Cannot load /usr/lib/apache2/modules/
libphp7.2.so into server: /usr/lib/apache2/modules/libphp7.2.so: cannot open shared object file: No such file or dire
ctory
Apr 14 18:16:42 php-fix-lp1865218-mod-php-upgrade apachectl[56492]: Action 'start' failed.
Apr 14 18:16:42 php-fix-lp1865218-mod-php-upgrade apachectl[56492]: The Apache error log may have more information.
Apr 14 18:16:42 php-fix-lp1865218-mod-php-upgrade systemd[1]: apache2.service: Control process exited, code=exited, s
tatus=1/FAILURE
Apr 14 18:16:42 php-fix-lp1865218-mod-php-upgrade systemd[1]: apache2.service: Failed with result 'exit-code'.
Apr 14 18:16:42 php-fix-lp1865218-mod-php-upgrade systemd[1]: Failed to start The Apache HTTP Server.
invoke-rc.d: release upgrade in progress, error is not fatal
Setting up fwupd-signed (1.27+1.3.9-4) ...
S...

It can be worked around by manual configuration:

# a2dismod php7.2
Module php7.2 disabled.
To activate the new configuration, you need to run:
  systemctl restart apache2

# sudo a2enmod php7.4
Considering dependency mpm_prefork for php7.4:
Considering conflict mpm_event for mpm_prefork:
Considering conflict mpm_worker for mpm_prefork:
Module mpm_prefork already enabled
Considering conflict php5 for php7.4:
Enabling module php7.4.
To activate the new configuration, you need to run:
  systemctl restart apache2

# apachectl -t
Syntax OK

# sudo service apache2 restart
sudo: setrlimit(RLIMIT_CORE): Operation not permitted

# ls /etc/apache2/mods-enabled/php*
/etc/apache2/mods-enabled/php7.4.conf /etc/apache2/mods-enabled/php7.4.load

# apache2ctl -S
VirtualHost configuration:
*:80 php-fix-lp1865218-mod-php-upgrade.lxd (/etc/apache2/sites-enabled/000-default.conf:1)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex watchdog-callback: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex mpm-accept: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33

I reproduced same issue on eoan->focal upgrades.

On the eoan->focal upgrade, removing and reinstalling mod-php reproduces the error:

# apt-get remove libapache2-mod-php7.4 libapache2-mod-php
...

# apt-get install libapache2-mod-php7.4 libapache2-mod-php
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  php-pear
The following NEW packages will be installed:
  libapache2-mod-php libapache2-mod-php7.4
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 1367 kB of archives.
After this operation, 4681 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libapache2-mod-php7.4 amd64 7.4.3-4ubuntu1 [1364 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libapache2-mod-php all 2:7.4+75 [2836 B]
Fetched 1367 kB in 1s (921 kB/s)
Selecting previously unselected package libapache2-mod-php7.4.
(Reading database ... 32021 files and directories currently installed.)
Preparing to unpack .../libapache2-mod-php7.4_7.4.3-4ubuntu1_amd64.deb ...
Unpacking libapache2-mod-php7.4 (7.4.3-4ubuntu1) ...
Selecting previously unselected package libapache2-mod-php.
Preparing to unpack .../libapache2-mod-php_2%3a7.4+75_all.deb ...
Unpacking libapache2-mod-php (2:7.4+75) ...
Setting up libapache2-mod-php7.4 (7.4.3-4ubuntu1) ...
libapache2-mod-php7.4: php7.3 module already enabled, not enabling PHP 7.4
Setting up libapache2-mod-php (2:7.4+75) ...
Processing triggers for libapache2-mod-php7.4 (7.4.3-4ubuntu1) ...

These look like the same problem, reported to Debian:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=893481
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911738

The error in question occurs during the postinst for /var/lib/dpkg/info/libapache2-mod-php7.4, where it is following this logic:

        PHP_MODULE=$(a2query -m | sed -n 's/^\(php[\.0-9]*\) (enabled.*)/\1/p')
        if [ -n "$PHP_MODULE" -a "$PHP_MODULE" != "php7.4" ]; then
            apache2_msg "err" "$DPKG_MAINTSCRIPT_PACKAGE: $PHP_MODULE module already enabled, not enabling PHP 7.4"
            return 1
        fi

I wonder if instead of erroring at this point, it instead ran dismod on the other modules, if it would resolve this problem. I.e., something more like:

        PHP_MODULE=$(a2query -m | sed -n 's/^\(php[\.0-9]*\) (enabled.*)/\1/p')
        if [ -n "$PHP_MODULE" -a "$PHP_MODULE" != "php7.4" ]; then
            apache2_msg "info" "$DPKG_MAINTSCRIPT_PACKAGE: Disabling $PHP_MODULE in favor of using PHP 7.4"
            apache2_invoke dismod $PHP_MODULE || exit 1
        fi

A PPA with the proposed fix is here:

    https://launchpad.net/~bryce/+archive/ubuntu/php-defaults-fix-lp1865218-mod-php-upgrade

Review and preliminary testing indicates this is ready.

One limitation of the proposed solution for Ubuntu is that this assumes a single supported PHP version for a given distro release. This will always hold true for Ubuntu, but not necessarily for Debian. For this reason, while I think this change is good for Ubuntu it may not be for Debian. However, once it's validated for Ubuntu I'm hopeful it can lead to a more generally compatible solution that works for both distros. But this should be good for a Ubuntu-only SRU.

This bug was fixed in the package php7.4 - 7.4.3-4ubuntu3

---------------
php7.4 (7.4.3-4ubuntu3) groovy; urgency=medium

  * libapache2-mod-php.postinst.extra: Disable other mod-php versions.
    Fixes failure when upgrading from previous versions of mod-php.
    (LP: #1865218)

 -- Bryce Harrington <email address hidden> Tue, 21 Apr 2020 23:04:30 +0000

Hello Simon, or anyone else affected,

Accepted php7.4 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php7.4/7.4.3-4ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Simon, or anyone else affected,

Accepted php7.3 into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php7.3/7.3.11-0ubuntu0.19.10.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Simon, or anyone else affected,

Accepted php7.2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php7.2/7.2.24-0ubuntu0.18.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted php7.4 (7.4.3-4ubuntu2.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

php-luasandbox/3.0.3-2build2 (arm64)
php-excimer/1.0.0~git20190913.d82eaf7-1build1 (s390x)
libsoup2.4/2.70.0-1 (amd64)
php-parser/4.2.2-2 (armhf, s390x, ppc64el, arm64, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#php7.4

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted php7.3 (7.3.11-0ubuntu0.19.10.5) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

php-horde-db/2.4.0-3ubuntu1 (i386, arm64, armhf, s390x, amd64, ppc64el)
kopano-webapp/3.5.9+dfsg1-1build1 (i386, arm64, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#php7.3

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted php7.2 (7.2.24-0ubuntu0.18.04.5) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

phpcpd/unknown (armhf)
php-horde-xml-element/unknown (armhf)
pkg-php-tools/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#php7.2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

I tried various things to validate either the package from -proposed or from Bryce's PPA but to no avail.

The problem is that do-release-upgrade doesn't let me use $distro-proposed despite using do-release-upgrade -p and Bryce's PPA is deactivated during the upgrade.

@bryce, I am sorry but I don't know how to verify what landed in -proposed (or even your PPA).

All attempts with do-release-update failed and I ended up filling LP: #1880760 to ask for an easy way to validate packages in the $distro-proposed.

Thanks for filing the bug report about the testing difficulty, I'm not sure I know of an easy workaround to that.

There's another way to do the upgrade, which is a bit more brute force, but should exercise the post-inst changes adequately which may be sufficient for sru validation:

1. Set up the earlier release, and install the PPA and fixed mod_php.
2. Manually edit /etc/apt/sources.list and change all instances of the earlier release to the new one. IOW, do this:
    $ sudo sed -i 's/xenial/bionic/g' /etc/apt/sources.list
3. Do the same for anything in /etc/apt/sources.list.d/. In particular, update the file for the PPA with the php7.4 fixes.
4. Manually upgrade
    $ sudo apt update && sudo apt -y dist-upgrade
    Note this may present different config prompts and such than the do-release-upgrade command.
5. Validate upgrade works, as per the original test case.

I haven't tested this process, but used to use it long ago, and it appears to still be the method used with debian, so I think it should work. (I wouldn't attempt this on production hardware, obviously.)

This bug was fixed in the package php7.2 - 7.2.24-0ubuntu0.18.04.6

---------------
php7.2 (7.2.24-0ubuntu0.18.04.6) bionic-security; urgency=medium

  * SECURITY UPDATE: Denial of service through oversized memory allocated
    - debian/patches/CVE-2019-11048.patch: changes types int to size_t
      in main/rfc1867.c.
    - CVE-2019-11048

 -- <email address hidden> (Leonidas S. Barbosa) Tue, 26 May 2020 10:09:11 -0300

This bug was fixed in the package php7.3 - 7.3.11-0ubuntu0.19.10.6

---------------
php7.3 (7.3.11-0ubuntu0.19.10.6) eoan-security; urgency=medium

  * SECURITY UPDATE: Denial of service through oversized memory allocated
    - debian/patches/CVE-2019-11048.patch: changes types int to size_t
      in main/rfc1867.c.
    - CVE-2019-11048

 -- <email address hidden> (Leonidas S. Barbosa) Tue, 26 May 2020 09:28:47 -0300

This bug was fixed in the package php7.4 - 7.4.3-4ubuntu2.2

---------------
php7.4 (7.4.3-4ubuntu2.2) focal-security; urgency=medium

  * SECURITY UPDATE: Denial of service through oversized memory allocated
    - debian/patches/CVE-2019-11048.patch: changes types int to size_t
      in main/rfc1867.c.
    - CVE-2019-11048

 -- <email address hidden> (Leonidas S. Barbosa) Tue, 26 May 2020 09:24:22 -0300

The cat's out of the bag already thanks to the releases to $distro-security keeping your patch! I did a verification nevertheless as it's now easier with the package being readily available so no need to use the trick in comment 33 (will keep it in mind for the future though, thanks Bryce!).

So upgrading Bionic (7.2.24-0ubuntu0.18.04.6) -> Focal (7.4.3-4ubuntu2.2)

libapache2-mod-php7.4: Disabling old php7.2 in favor of using PHP 7.4
Setting up apache2 (2.4.41-4ubuntu3) ...
info: Executing deferred 'a2dismod php7.2' for package libapache2-mod-php7.4
Module php7.2 disabled.
info: Executing deferred 'a2enmod php7.4' for package libapache2-mod-php7.4
Enabling module php7.4.
Setting up libapache2-mod-php (2:7.4+75) ...

Thanks you very much Bryce.

Thanks Simon, that is indeed good news, and thanks for verifying. I'll go ahead and mark the sru tasks completed.

Note for the future: do-release-upgrade can be used with --allow-third-party which will keep the ppa enabled and also switch it to fetch packages from the new release.

On 2020-05-29 9:15 a.m., Andreas Hasenack wrote:
> Note for the future: do-release-upgrade can be used with --allow-third-
> party which will keep the ppa enabled and also switch it to fetch
> packages from the new release.

Ah, thanks! I will try to enable $distro-proposed as it it was a
third-party repo and see how it goes ;)

On 2020-05-29 9:27 a.m., Simon Déziel wrote:
> On 2020-05-29 9:15 a.m., Andreas Hasenack wrote:
>> Note for the future: do-release-upgrade can be used with --allow-third-
>> party which will keep the ppa enabled and also switch it to fetch
>> packages from the new release.
>
> Ah, thanks! I will try to enable $distro-proposed as it it was a
> third-party repo and see how it goes ;)

No luck:

$ cat /etc/apt/sources.list.d/proposed.list
# deb http://archive.ubuntu.com/ubuntu focal-proposed main universe #Not
for humans during development stage of release focal

Maybe it handles proposed differently. I have used it with ppas only.

I had also wondered if --allow-third-party (or its corresponding env var RELEASE_UPGRADER_ALLOW_THIRD_PARTY) would be worth trying, however in looking at the do-release-upgrader code, in DistUpgradeController::rewriteSourcesList() it looks like it's used for skipping the mirror check. So if you're an admin with a local mirror of the Ubuntu repositories this would allow use of your mirror when your users are upgrading. Something like that.

Handling of proposed is special cased separately in DistUpgradeController::rewriteSourcesList():

            # Disable proposed on upgrade to a development release.
            if (not entry.disabled and self.options
                and self.options.devel_release == True and
                "%s-proposed" % self.fromDist in entry.dist):
                logging.debug("upgrade to development release, disabling proposed")
                entry.dist = "%s-proposed" % self.toDist
                entry.comment += _("Not for humans during development stage of release %s") % self.toDist
                entry.disabled = True
                continue

I had some thoughts on how this could be modified to support a cli param + env var like --allow-third-parties, but I'll write those notes for the bug Simon opened, LP: #1880760.