Ubuntu
ca-certificates package

Distrust "DST Root CA X3"

Bug #1944481 reported by Dimitri John Ledkov
260
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ca-certificates (Debian)
Fix Released
Unknown
ca-certificates (Ubuntu)
Fix Released
Undecided
Marc Deslauriers
Trusty
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Marc Deslauriers
Focal
Fix Released
Undecided
Marc Deslauriers
Hirsute
Fix Released
Undecided
Marc Deslauriers
Impish
Fix Released
Undecided
Marc Deslauriers

Bug Description

[Impact]

 * ca-certificates trusts the letsencrypt CA certificate "ISRG Root X1"
 * ca-certificates also trusts the CA certificate "DST Root CA X3" which cross-signs letencrypt CA
 * "DST Root CA X3" is about to expire, however it has issued an updated cross-signature to letsencrypt beyond its own expiry
 * This causes issues with older implementations of openssl & gnutls that reject such chains when offered to clients by servers.
 * We have provided fixes for openssl in xenial and gnutls in bionic/xenial, however trusty systems remain affected. Also any self built old copies of openssl/gnutls remain suspeptible to this expiry.
 * One solution is to blacklist the "DST Root CA X3" from the ca-certificates package as described at https://blog.devgenius.io/rhel-centos-7-fix-for-lets-encrypt-change-8af2de587fe4 - connectivity to sites chained to "DST Root CA X3" will be unaffected, and servers that chain to both "ISRG Root X1" and "DST Root CA X3" should start to work unmodified.
 * This is similar to how this was handled for AddTrust before

"* mozilla/blacklist.txt: blacklist expired AddTrust External Root CA."

[Test Plan]

 * Install old/current ca-certificates faketime wget curl libcurl3-gnutls

# faketime 2021-10-01 wget https://pskov.surgut.co.uk
--2021-10-01 00:00:00-- https://pskov.surgut.co.uk/
Resolving pskov.surgut.co.uk (pskov.surgut.co.uk)... 2a01:4f8:c17:3dd8::1, 49.12.37.5
Connecting to pskov.surgut.co.uk (pskov.surgut.co.uk)|2a01:4f8:c17:3dd8::1|:443... connected.
ERROR: cannot verify pskov.surgut.co.uk's certificate, issued by '/C=US/O=Let\'s Encrypt/CN=R3':
  Issued certificate has expired.
To connect to pskov.surgut.co.uk insecurely, use `--no-check-certificate'.

# LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 faketime 2021-10-01 curl https://pskov.surgut.co.uk >/dev/null
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (60) SSL certificate problem: certificate has expired

 * Install new ca-certificates package

# faketime 2021-10-01 wget https://pskov.surgut.co.uk
--2021-10-01 00:00:00-- https://pskov.surgut.co.uk/
Resolving pskov.surgut.co.uk (pskov.surgut.co.uk)... 2a01:4f8:c17:3dd8::1, 49.12.37.5
Connecting to pskov.surgut.co.uk (pskov.surgut.co.uk)|2a01:4f8:c17:3dd8::1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 612 [text/html]
Saving to: 'index.html.3'

100%[====================================================>] 612 --.-K/s in 0s

2021-10-01 00:00:00 (71.7 MB/s) - 'index.html.3' saved [612/612]

 LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 faketime 2021-10-01 curl https://pskov.surgut.co.uk >/dev/null
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 612 100 612 0 0 5794 0 --:--:-- --:--:-- --:--:-- 5828

Download is successful.

[Where problems could occur]

 * Connectivity to "DST Root CA X3" websites only, even under faketime set to dates prior to 30th of September 2021 will not work, as "DST Root CA X3" certificate is no longer installed. users should locally install and enable that CA certificate, or allow dangerous unverified connectivity to websites using expired CA certs.

[Other Info]

 * Related openssl and gnutls28 bugs are https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1928989 and https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1928648

See original description
Tags: patch
Dimitri John Ledkov (xnox)
description: updated
description: updated
Dimitri John Ledkov (xnox)
description: updated
Revision history for this message
Dimitri John Ledkov (xnox) wrote :
description: updated
Dimitri John Ledkov (xnox)
description: updated
description: updated
Revision history for this message
Dimitri John Ledkov (xnox) wrote :
Marc Deslauriers (mdeslaur)
information type: Private Security → Public Security
Changed in ca-certificates (Ubuntu Bionic):
assignee: nobody → Marc Deslauriers (mdeslaur)
Changed in ca-certificates (Ubuntu Focal):
assignee: nobody → Marc Deslauriers (mdeslaur)
Changed in ca-certificates (Ubuntu Hirsute):
assignee: nobody → Marc Deslauriers (mdeslaur)
Changed in ca-certificates (Ubuntu Impish):
assignee: nobody → Marc Deslauriers (mdeslaur)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ca-certificates - 20210119~20.04.2

---------------
ca-certificates (20210119~20.04.2) focal-security; urgency=medium

  [ Dimitri John Ledkov ]
  * mozilla/blacklist.txt: blacklist expired "DST Root CA X3".
    (LP: #1944481)

 -- Marc Deslauriers <email address hidden> Wed, 22 Sep 2021 07:46:54 -0400

Changed in ca-certificates (Ubuntu Focal):
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ca-certificates - 20210119~18.04.2

---------------
ca-certificates (20210119~18.04.2) bionic-security; urgency=medium

  [ Dimitri John Ledkov ]
  * mozilla/blacklist.txt: blacklist expired "DST Root CA X3".
    (LP: #1944481)

 -- Marc Deslauriers <email address hidden> Wed, 22 Sep 2021 07:46:54 -0400

Changed in ca-certificates (Ubuntu Bionic):
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ca-certificates - 20210119ubuntu0.21.04.1

---------------
ca-certificates (20210119ubuntu0.21.04.1) hirsute-security; urgency=medium

  [ Dimitri John Ledkov ]
  * mozilla/blacklist.txt: blacklist expired "DST Root CA X3".
    (LP: #1944481)

 -- Marc Deslauriers <email address hidden> Wed, 22 Sep 2021 07:46:54 -0400

Changed in ca-certificates (Ubuntu Hirsute):
status: New → Fix Released
Ubuntu Foundations Team Bug Bot (crichton)
tags: added: patch
Marc Deslauriers (mdeslaur)
Changed in ca-certificates (Ubuntu Impish):
status: New → Fix Committed
Changed in ca-certificates (Ubuntu Trusty):
status: New → Fix Released
Changed in ca-certificates (Ubuntu Xenial):
status: New → Fix Released
Romain Couturat (romaincout)
information type: Public Security → Private Security
Romain Couturat (romaincout)
information type: Private Security → Public
information type: Public → Public Security
Revision history for this message
Matt Jones (mattjones86) wrote :

I ran into an SSL verification issue today, caused by this change.

It seems that some older LetsEncrypt clients have still recently been issuing valid certificates signed by the DST Root CA X3 root.

These certificates would have otherwise continued to work normally until the root expired (September 30th 2021), but have been distrusted early due to this change. (Indeed the certificate in question in my case was still trusted by the latest Chrome etc.)

The best fix is to make sure the ACME client is up-to-date and re-issue the certificates under the new root cert.

Posting for awareness - surprised I'm the first!

Revision history for this message
Joel Sing (jsing) wrote :

@mattjones86 that does not seem expected - Let's Encrypt have been issuing certificate from their R3 intermediate since December 2021 (https://community.letsencrypt.org/t/beginning-issuance-from-r3/139018) and have been supplying two intermediates (an Let's Encrypt R3 to ISRG Root X1 and a Let's Encrypt R3 to DST Root CA X3) in the default chain since 4th May 2021 (https://community.letsencrypt.org/t/production-chain-changes/150739). Given that certificates issued by Let's Encrypt have a maximum validity period of 90 days, all certificates that are still valid after the 4th of August would have been issued in this manner.

The only thing I could think of that would explain the behaviour mentioned, is if your ACME client was failing to update the certificate chain/bundle (or your server was configured to serve and old/stale bundle). Most browsers (including Chrome) will also automatically fetch issuer intermediate certificates if they're not supplied by the server.

Revision history for this message
Matt Jones (mattjones86) wrote :

@jsing You may well be correct that the server was incorrectly configured, unfortunately it was a Windows server managed by a third party and I don't know precisely how it was set up. Given that the cert in question was issued on 9th September 2021 I suspect it was a misconfiguration of their intermediate cert they were sending.

Revision history for this message
Collin Anderson (cmawebsite) wrote (last edit ):

Yes, I'm running into the issue above, where a windows IIS server is not correctly serving the new Let's Encrypt R3 intermediate certificate chain, signed by "ISRG Root X1", and still serving the old R3 intermediate signed by "DST Root CA X3" (which means it's going to fail for everyone else on Sept 30th.) Windows IIS server might need an update or might need to be rebooted. https://community.certifytheweb.com/t/upcoming-expiry-of-dst-root-ca-x3-and-r3-intermediate-for-lets-encrypt/1480

In the meantime, from the ubuntu point of view, how do I roll this update back? The cert is still valid for another week. `sudo apt install ca-certificates=20210119~20.04.1` says `E: Version '20210119~20.04.1' for 'ca-certificates' was not found`.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

You can find older packages on the "full publishing history" from launchpad:

https://launchpad.net/ubuntu/+source/ca-certificates/+publishinghistory

You can either download it manually or use the pull-lp-debs(1) command from the ubuntu-dev-tools package.

Thanks

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ca-certificates - 20210119ubuntu1

---------------
ca-certificates (20210119ubuntu1) impish; urgency=medium

  [ Dimitri John Ledkov ]
  * mozilla/blacklist.txt: blacklist expired "DST Root CA X3".
    (LP: #1944481)

 -- Marc Deslauriers <email address hidden> Wed, 22 Sep 2021 07:46:54 -0400

Changed in ca-certificates (Ubuntu Impish):
status: Fix Committed → Fix Released
Jeremy Bícha (jbicha)
no longer affects: ca-certificates (Fluxbuntu)
Bug Watch Updater (bug-watch-updater)
Changed in ca-certificates (Debian):
status: Unknown → Fix Released
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.