libssl 1.0.2g-1ubuntu4.4 and 1.0.1f-1ubuntu2.20 cause PHP SSL cert validation to segfault

The primary issue is some patch in the latest openssl, which breaks current php7.0. Not any change in the PHP package.

StacktraceTop:
 strlen () at ../sysdeps/x86_64/strlen.S:106
 add_assoc_string_ex (arg=arg@entry=0x7f19df018cf0, key=key@entry=0x55ab940bbf59 "serialNumber", key_len=key_len@entry=12, str=0x0) at /build/php7.0-lPMnpS/php7.0-7.0.8/Zend/zend_API.c:1390
 zif_openssl_x509_parse (execute_data=<optimized out>, return_value=0x7f19df018cf0) at /build/php7.0-lPMnpS/php7.0-7.0.8/ext/openssl/openssl.c:2017
 dtrace_execute_internal (execute_data=<optimized out>, return_value=<optimized out>) at /build/php7.0-lPMnpS/php7.0-7.0.8/Zend/zend_dtrace.c:107
 ZEND_DO_FCALL_SPEC_HANDLER () at /build/php7.0-lPMnpS/php7.0-7.0.8/Zend/zend_vm_execute.h:844

The stacktrace would seem to indicate that libssl indeed returned a null string here, from i2s_ASN1_INTEGER(NULL, X509_get_serialNumber(cert))

Relevant php7.0 code here:

https://github.com/php/php-src/blob/f13fd9e72a13e80512f6c8b2302e42d4f252c479/ext/openssl/openssl.c#L2295

The issue is not limited to Ubuntu 16.04 and PHP 7.

We experience a similar issue on Ubuntu 14.04 using PHP 5.5 (se exact system info below).

Tonight's unattended openssl update from 1.0.1f-1ubuntu2.19 to 1.0.1f-1ubuntu2.20 (http://www.ubuntu.com/usn/usn-3087-1/, http://changelogs.ubuntu.com/changelogs/pool/main/o/openssl/openssl_1.0.1f-1ubuntu2.20/changelog) causes our Satis installation (https://github.com/composer/satis) to segfault on trying to establish HTTPS connections;

Start-Date: 2016-09-23 04:45:30
Upgrade: libssl1.0.0:amd64 (1.0.1f-1ubuntu2.19, 1.0.1f-1ubuntu2.20), libssl-dev:amd64 (1.0.1f-1ubuntu2.19, 1.0.1f-1ubuntu2.20), libssl-doc:amd64 (1.0.1f-1ubuntu2.19, 1.0.1f-1ubuntu2.20), openssl:amd64 (1.0.1f-1ubuntu2.19, 1.0.1f-1ubuntu2.20)
End-Date: 2016-09-23 04:45:34

We have isolated it to this simple php command trying to parse the openssl provided ca-certs also triggering the issue;

# php -r "openssl_x509_parse(file_get_contents('/etc/ssl/certs/ca-certificates.crt'));"
Segmentation fault (core dumped)

Downgrading is only possible to 1.0.1f-1ubuntu2, which causes some dependencies to be uninstalled which seems counterproductive;

# apt-get install libssl1.0.0=1.0.1f-1ubuntu2.19
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '1.0.1f-1ubuntu2.19' for 'libssl1.0.0' was not found

# apt-get install libssl1.0.0=1.0.1f-1ubuntu2
...
The following packages will be REMOVED:
  libssl-dev node-gyp nodejs-dev npm php5-dev
The following packages will be DOWNGRADED:
  libssl1.0.0
0 upgraded, 0 newly installed, 1 downgraded, 5 to remove and 1 not upgraded.

Why would those dependencies be removed and why can't I pinpoint that I want 1.0.1f-1ubuntu2.19 installed?

# php -v
PHP 5.5.9-1ubuntu4.19 (cli) (built: Jul 28 2016 19:31:33)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies
    with Xdebug v2.2.3, Copyright (c) 2002-2013, by Derick Rethans

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty

Status changed to 'Confirmed' because the bug affects multiple users.

@mikini, I actually had a similar situation with apt-get wanting to remove npm. That's due to npm depending on node-gyp, which depends on nodejs-dev, which depends on libssl-dev. You need to install an old version of that package as well, you can't have a new libssl-dev package and an old libssl package installed together.

So something like:

apt-get install libssl1.0.0=1.0.1f-1ubuntu2 libssl-dev=1.0.1f-1ubuntu2 npm node-gyp nodejs-dev

should ensure you'll get compatible older versions installed, and still have the Node.js stuff.

However, that 1.0.1f-1ubuntu2 version seems quite old and could contain lots of vulnerabilities... I'd be wary of using it unless your server won't be doing SSL termination for clients from untrusted sources. Either because you SSL terminate at a load balancer, a reverse proxy or the like, or because your server is only accessible from a private network, like mine.

A better option would be to try and source the libssl and libssl-dev binaries for the immediately preceding 1.0.1f-1ubuntu2.19 version from somewhere else.

Thanks @ollisa.

I had the same thoughts about 1.0.1f-1ubuntu2 so I found a downloadable build at https://launchpad.net/ubuntu/+source/openssl/1.0.1f-1ubuntu2.19. Installing just the ubuntu2.19 version of libssl1.0.0 solved the issue;

wget https://launchpad.net/~ubuntu-security/+archive/ubuntu/ppa/+build/9679884/+files/libssl1.0.0_1.0.1f-1ubuntu2.19_amd64.deb
dpkg -i libssl1.0.0_1.0.1f-1ubuntu2.19_amd64.deb

Now the certs can be parsed without segfault;
# php -r "echo gettype(openssl_x509_parse(file_get_contents('/etc/ssl/certs/ca-certificates.crt')));"
array

A good idea would be to put the package on hold to prevent further automatic upgrades. Though you'd then need to manually verify and unhold when a fix is out

# apt-mark hold libssl1.0.0
libssl1.0.0 set on hold.

I can reproduce this and will release an updated openssl package today.

Also affected 1.0.1-4ubuntu5.37 on 12.04

Can confirm that this affects 1.0.1-4ubuntu5.37 on 12.04

Reproducible by trying to openssl_x509_parse the ssl cert for sourceforge with PHP 5.5.30-1+deb.sury.org~precise+1

$ openssl s_client -connect sourceforge.net:443 </dev/null |& sed -n '/BEGIN CERTIFICATE/,$p' | sed '/END CERTIFICATE/q' > cert.txt
$ echo "<?php openssl_x509_parse(file_get_contents('cert.txt'));" > segfault.php
$ php segfault.php
Segmentation fault (core dumped)

The backtrace:
$ gdb php
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /usr/bin/php...(no debugging symbols found)...done.
(gdb) r segf.php
Starting program: /usr/bin/php segf.php
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff5c40f81 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff5c40f81 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00000000006e8e8d in add_assoc_string_ex ()
#2 0x00000000004a67ba in zif_openssl_x509_parse ()
#3 0x00000000006d4959 in dtrace_execute_internal ()
#4 0x00000000007911de in ?? ()
#5 0x0000000000754358 in execute_ex ()
#6 0x00000000006d4846 in dtrace_execute_ex ()
#7 0x00007ffff4f72ecc in ?? () from /usr/lib/php5/20121212/ioncube_loader_lin_5.5.so
#8 0x00000000006e66b4 in zend_execute_scripts ()
#9 0x000000000068380d in php_execute_script ()
#10 0x00000000007949c3 in ?? ()
#11 0x0000000000465081 in main ()
(gdb)

Packages that fix this issue are currently being built in the security team PPA:

https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+packages

They will be published as soon as they finish building and have gone through QA.

This bug was fixed in the package openssl - 1.0.1-4ubuntu5.38

---------------
openssl (1.0.1-4ubuntu5.38) precise-security; urgency=medium

  * SECURITY REGRESSION: incomplete fix for CVE-2016-2182 (LP: #1626883)
    - debian/patches/CVE-2016-2182-2.patch: fix off-by-one in overflow
      check in crypto/bn/bn_print.c.

 -- Marc Deslauriers <email address hidden> Fri, 23 Sep 2016 07:59:32 -0400

This bug was fixed in the package openssl - 1.0.1f-1ubuntu2.21

---------------
openssl (1.0.1f-1ubuntu2.21) trusty-security; urgency=medium

  * SECURITY REGRESSION: incomplete fix for CVE-2016-2182 (LP: #1626883)
    - debian/patches/CVE-2016-2182-2.patch: fix off-by-one in overflow
      check in crypto/bn/bn_print.c.

 -- Marc Deslauriers <email address hidden> Fri, 23 Sep 2016 07:57:00 -0400

This bug was fixed in the package openssl - 1.0.2g-1ubuntu4.5

---------------
openssl (1.0.2g-1ubuntu4.5) xenial-security; urgency=medium

  * SECURITY REGRESSION: incomplete fix for CVE-2016-2182 (LP: #1626883)
    - debian/patches/CVE-2016-2182-2.patch: fix off-by-one in overflow
      check in crypto/bn/bn_print.c.

 -- Marc Deslauriers <email address hidden> Fri, 23 Sep 2016 08:00:13 -0400

Thank you. I can verify libssl1.0.0 1.0.2g-1ubuntu4.5 no longer exhibits the crash:

jenkins@ubuntutemplate:/var/lib/jenkins/workspace/imt-erp-e2e-flaky/webshop/vagrant/wordpress$ apt-cache policy libssl1.0.0
libssl1.0.0:
  Installed: 1.0.2g-1ubuntu4.5
  Candidate: 1.0.2g-1ubuntu4.5
  Version table:
 *** 1.0.2g-1ubuntu4.5 500
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.2g-1ubuntu4.2 500
        500 http://fi.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     1.0.2g-1ubuntu4 500
        500 http://fi.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
jenkins@ubuntutemplate:/var/lib/jenkins/workspace/imt-erp-e2e-flaky/webshop/vagrant/wordpress$ DATABASE_DATABASE=wordpressmastere2e wp plugin install --force --activate wp-cfm
Deprecated: Methods with the same name as their class will not be constructors in a future version of PHP; WP_Import has a deprecated constructor in /var/lib/jenkins/workspace/imt-erp-e2e-flaky/webshop/vagrant/wordpress/wp-content/plugins/wordpress-importer/wordpress-importer.php on line 38
Notice: Undefined offset: 4 in phar:///usr/local/bin/wp/php/WP_CLI/DocParser.php on line 124
Installing WP-CFM (1.4.5)
Ladataan pakettia lähteestä https://downloads.wordpress.org/plugin/wp-cfm.zip...
Using cached file '/home/jenkins/.wp-cli/cache/plugin/wp-cfm-1.4.5.zip'...
Puretaan pakettia...
Asennetaan lisäosaa...
Poistetaan lisäosan vanhaa versiota...
Lisäosa päivitetty onnistuneesti.
Activating 'wp-cfm'...
Warning: Plugin 'wp-cfm' is already active.
jenkins@ubuntutemplate:/var/lib/jenkins/workspace/imt-erp-e2e-flaky/webshop/vagrant/wordpress$

Thanks for the fix.

I too can verify that our system doesn't segfault on Ubuntu 14.04 (trusty) using latest libssl1.0.0 (=1.0.1f-1ubuntu2.21);

# dpkg -l |grep libssl1.0.0
ii libssl1.0.0:amd64 1.0.1f-1ubuntu2.21 amd64 Secure Sockets Layer toolkit - shared libraries

# php -r "echo gettype(openssl_x509_parse(file_get_contents('/etc/ssl/certs/ca-certificates.crt')));"
array

We'll definitely be reconsidering which systems will be applying security upgrades unattended in the future.

This experience makes me wonder how patches for the -security suites (default for unattended-upgrades) are tested and QA'ed. Can anything be done to the Ubuntu process to prevent things like this happening again?

I'm unfamiliar with how this is done currently so excuse my ignorance. But I'm wondering why there seem to be no collaboration or correlation between Ubuntu and Debian security updates. Debian seems to have got this one right in the first shot (DSA is here https://www.debian.org/security/2016/dsa-3673).

BTW: the links to upstream patches on the Ubuntu CVE page (http://people.canonical.com/~ubuntu-security/cve/2016/CVE-2016-2182.html) are invalid caused by a version string being appended to the commit hash (looks like borked wiki syntax).

> This experience makes me wonder how patches for the -security suites (default for unattended-upgrades) are tested and QA'ed. Can anything be done to the Ubuntu process to prevent things like this happening again?

For OpenSSL, we run it through a test suite and also test it with commonly run software such as Apache, Wget, etc. In this instance, the issue was an off-by-one which means it only affected certain certificates, and unfortunately not the certs that were used in our test suite. We've now added a test to parse all certs in the ca-certificates.crt file so this particular issue doesn't happen again.

> Debian seems to have got this one right in the first shot (DSA is here https://www.debian.org/security/2016/dsa-3673).

Debian hit the very same regression. See https://lists.debian.org/debian-security-announce/2016/msg00255.html

> BTW: the links to upstream patches on the Ubuntu CVE page (http://people.canonical.com/~ubuntu-security/cve/2016/CVE-2016-2182.html) are invalid caused by a version string being appended to the commit hash

Thanks, I'll get that fixed.