Ubuntu
openssl package

c_rehash creating bogus links to ca-certificates.crt

Bug #854927 reported by Scott Moser
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
ca-certificates (Debian)
Fix Released
Unknown
ca-certificates (Ubuntu)
Fix Released
High
Loïc Minier
Oneiric
Fix Released
High
Loïc Minier
openssl (Ubuntu)
Fix Released
High
Loïc Minier
Ubuntu ubuntu-11.10
Oneiric
Fix Released
High
Loïc Minier
Ubuntu ubuntu-11.10

Bug Description

$ wget https://www.google.com
--2011-09-20 18:12:46-- https://www.google.com/
Resolving www.google.com... 209.85.169.105, 209.85.169.106, 209.85.169.147, ...
Connecting to www.google.com|209.85.169.105|:443... connected.
ERROR: cannot verify www.google.com's certificate, issued by `/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA':
  Unable to locally verify the issuer's authority.
To connect to www.google.com insecurely, use `--no-check-certificate'.

$ curl -sS https://launchpad.net
curl: (35) error:0B07C065:x509 certificate routines:X509_STORE_add_cert:cert already in hash table

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: openssl 1.0.0e-2ubuntu1
ProcVersionSignature: User Name 3.0.0-11.18-virtual 3.0.4
Uname: Linux 3.0.0-11-virtual i686
ApportVersion: 1.23-0ubuntu1
Architecture: i386
Date: Tue Sep 20 18:11:11 2011
Ec2AMI: ami-00000090
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: openssl
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Ben Howard (darkmuggle-deactivatedaccount) wrote :

Marking as confirmed.

Changed in openssl (Ubuntu):
status: New → Confirmed
Ben Howard (darkmuggle-deactivatedaccount)
Changed in openssl (Ubuntu):
importance: Undecided → High
Kate Stewart (kate.stewart)
tags: added: rls-mgr-o-tracking
Changed in openssl (Ubuntu Oneiric):
milestone: none → ubuntu-11.10-beta-2
Dustin Kirkland  (kirkland)
Changed in openssl (Ubuntu Oneiric):
status: Confirmed → In Progress
Revision history for this message
Steve Langasek (vorlon) wrote :

This has been traced to a broken hash directory:

11:40 < kirkland> lrwxrwxrwx 1 root root 19 2011-09-20 01:34 /usr/lib/ssl/certs/55a10908.0 -> ca-certificates.crt
11:40 < kirkland> -rw-r--r-- 1 root root 240312 2011-09-20 01:32 /usr/lib/ssl/certs/ca-certificates.crt

This is expected to point to the specific certificate file, ValiCert_Class_2_VA.pem, instead; but on new installs since the latest upload of the new upstream version of openssl, c_rehash is giving preference to the ca-certificates bundle file over the individual cert files, and libssl subsequently is unable to use ca-certificates.crt for certificate validation.

I would definitely say there's a bug in openssl here, since c_rehash shouldn't create symlinks that the library will be subsequently unable to use; but I think we can work around it in ca-certificates by just making sure the bundle file is moved out of the way at the time we're calling c_rehash - since any time we call c_rehash we're regenerating that bundle file anyway.

Changed in ca-certificates (Ubuntu Oneiric):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Steve Langasek (vorlon)
milestone: none → ubuntu-11.10-beta-2
Changed in openssl (Ubuntu Oneiric):
milestone: ubuntu-11.10-beta-2 → ubuntu-11.10
status: In Progress → Triaged
assignee: nobody → Colin Watson (cjwatson)
Revision history for this message
Steve Langasek (vorlon) wrote :

12:41 < sbeattie> slangasek: behavior change in c_rehash is likely due to the patch that fixed
                  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=628780

summary: - wget, curl can't verify certificates
+ c_rehash creating bogus links to ca-certificates.crt
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ca-certificates - 20110502+nmu1ubuntu2

---------------
ca-certificates (20110502+nmu1ubuntu2) oneiric; urgency=low

  * Really only call --fresh on upgrade, instead of all the time; thanks to
    Adam Conrad for catching this in the queue.

ca-certificates (20110502+nmu1ubuntu1) oneiric; urgency=low

  * sbin/update-ca-certificates: move the ca-certificates.crt bundle out of
    the way before calling c_rehash, so that symlinks don't accidentally get
    pointed here, breaking openssl certificate verification. LP: #854927.
  * debian/postinst: kludge in support for running
    update-ca-certificates --fresh on upgrade, to ensure we fix up the hash
    for anyone who happened to install from a daily.
 -- Steve Langasek <email address hidden> Tue, 20 Sep 2011 12:49:57 -0700

Changed in ca-certificates (Ubuntu Oneiric):
status: In Progress → Fix Released
Revision history for this message
Steve Beattie (sbeattie) wrote :

Following up on the irc comment that Steve Langasek pasted, I can confirm that reverting the patch http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/oneiric/openssl/oneiric/revision/58#debian/patches/c_rehash-multi.patch followed by update-ca-certificates --fresh (without the workaround Steve added) also corrects the hashing/verification issue. However, it does seem like the c_rehash patch is correcting undesirable behavior on its part.

Revision history for this message
Colin Watson (cjwatson) wrote :

Is this really the entirety of the bug? With the new openssl but the old ca-certificates, I ran:

  $ sudo update-ca-certificates --fresh
  ...
  $ ls -l /usr/lib/ssl/certs/55a10908.0
lrwxrwxrwx 1 root root 19 2011-09-21 13:27 /usr/lib/ssl/certs/55a10908.0 -> ca-certificates.crt
  $ curl -sS http://launchpad.net
  <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>301 Moved Permanently</title>
  </head><body>
  <h1>Moved Permanently</h1>
  <p>The document has moved <a href="https://launchpad.net/">here</a>.</p>
  <hr>
  <address>Apache/2.2.14 (Ubuntu) Server at launchpad.net Port 80</address>
  </body></html>

What am I missing? While we could certainly change c_rehash to make sure it always prefers .pem files over .crt (and that might be preferable anyway), I wonder why libssl is unable to deal with the .crt files ...

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

What exactly are you trying to show Colin? You're connecting to http...

Revision history for this message
Colin Watson (cjwatson) wrote :

Whoops, I'm also unwell today and not thinking clearly. But in any case HTTPS works too:

$ wget https://www.google.com
--2011-09-21 14:52:14-- https://www.google.com/
Resolving www.google.com... 209.85.147.147, 209.85.147.99, 209.85.147.103, ...
Connecting to www.google.com|209.85.147.147|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://encrypted.google.com/ [following]
--2011-09-21 14:52:14-- https://encrypted.google.com/
Resolving encrypted.google.com... 209.85.147.100, 209.85.147.101, 209.85.147.102, ...
Connecting to encrypted.google.com|209.85.147.100|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html'

    [ <=> ] 11,434 --.-K/s in 0.07s

2011-09-21 14:52:15 (165 KB/s) - `index.html' saved [11434]

$ curl -sS https://launchpad.net
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
[...]

Revision history for this message
Colin Watson (cjwatson) wrote :

Should be fixed by Loïc's recent change:

openssl (1.0.0e-2ubuntu2) oneiric; urgency=low

  * Unapply patch c_rehash-multi and comment it out in the series as it breaks
    parsing of certificates with CRLF line endings and other cases (see
    Debian #642314 for discussion), it also changes the semantics of c_rehash
    directories by requiring applications to parse hash link targets as files
    containing potentially *multiple* certificates rather than exactly one.
    LP: #855454.

 -- Loïc Minier <email address hidden> Tue, 27 Sep 2011 18:13:07 +0200

Changed in openssl (Ubuntu Oneiric):
assignee: Colin Watson (cjwatson) → Loïc Minier (lool)
status: Triaged → Fix Released
Revision history for this message
Loïc Minier (lool) wrote :

While this wont happen with current ca-certificates, I think we should revert the changes which caused this bug:
in Debian's 20110421 QA upload, a c_rehash call was added to postinst for upgrades from versions <= 20090814+nmu3, this was an attempt to rebuild the symlinks in /etc/ssl/certs, but because update-ca-certificates wasn't removing /etc/ssl/cert/ca-certificates.crt, it did generate one symlink to this file for the first certificate. With the Debian change from openssl 1.0.0e-1 to support multiple certificates in one file, this probably took even worse proportions. However this probably depended on the order in which c_rehash processed files; it just does readdir() and generates links for the first certificate of each .pem and .crt file it finds.

Now in 20110502+nmu1ubuntu1/20110502+nmu1ubuntu2, a call was added to properly regenerate the links, but kept the plain c_rehash call *after* it in the postinst, so that it might trigger when upgrading from <= 20090814+nmu3 (so upgrades from natty or lucid will cause this).

Because of the new call I've added in20110502+nmu1ubuntu4 to regenerates certs when upgrading from <= 20110502+nmu1ubuntu4, this should be fixed for oneiric users.

Now, what needs to be fixed:
* plain c_rehash is wrong in any case; also an issue in Debian (and the rm needs to be copied there too)
* postinst has tons of update-ca-certificates calls, mine is the strongest one as it affects all updates (from natty); all of these should be dropped after oneiric

Now this could be fixed in oneiric + 1, but it would be clearer to remove these now to prevent any regression when removing the postinst snippets (e.g. leaving the plain c_rehash call alone after oneiric would be wrong).

Changed in ca-certificates (Ubuntu Oneiric):
assignee: Steve Langasek (vorlon) → Loïc Minier (lool)
milestone: ubuntu-11.10-beta-2 → none
status: Fix Released → Triaged
Revision history for this message
Loïc Minier (lool) wrote :

I've sent a patch to Debian including Steve's changes to remove ca-certificates.crt before running c_rehash in update-ca-certificates; will set bug id once I have it.

Changed in ca-certificates (Ubuntu Oneiric):
status: Triaged → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ca-certificates - 20110502+nmu1ubuntu5

---------------
ca-certificates (20110502+nmu1ubuntu5) oneiric; urgency=low

  * Tweak postinst to not run update-ca-certificates multiple times and remove
    dangerous plain c_rehash snippet; LP: #854927.
 -- Loic Minier <email address hidden> Wed, 28 Sep 2011 15:49:34 +0200

Changed in ca-certificates (Ubuntu Oneiric):
status: Fix Committed → Fix Released
Loïc Minier (lool)
Changed in ca-certificates (Debian):
importance: Undecided → Unknown
status: New → Unknown
Bug Watch Updater (bug-watch-updater)
Changed in ca-certificates (Debian):
status: Unknown → New
Bug Watch Updater (bug-watch-updater)
Changed in ca-certificates (Debian):
status: New → Fix Committed
Bug Watch Updater (bug-watch-updater)
Changed in ca-certificates (Debian):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.