init: symbol lookup error: /lib/powerpc64le-linux-gnu/libnss_ldap.so.2: undefined symbol: __libc_lock_lock

Bug #1387594 reported by Louis Bouchard
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
libnss-ldap (Debian)
Fix Released
Unknown
libnss-ldap (Ubuntu)
Fix Released
Critical
Unassigned
Precise
Invalid
Critical
Unassigned
Trusty
Fix Released
Critical
Louis Bouchard
Utopic
Fix Released
Critical
Louis Bouchard

Bug Description

[SRU justification]
The version of the library in the archive for Utopic and Trusty has been built prior to a change in glibc that removes an expected symbol. Rebuild of the libnss-ldap library with the current source package will render the library unusable and may cause systems to become unbootable.

[Impact]
Without this fix, a rebuild of the libnss-ldap package will cause grave impact where usage of the libnss-ldap will segfault. Such a segfault during the boot process will cause the system to become unbootable.

[Fix]
Backport the glibc-2.16.patch that is now merged in Vivid into Utopic and Trusty. The version of the library in Precise already uses the correct glibc symbol.

[Test Case]
On a server properly configured for ldap authentication :

root@trusty-ldapclient:~# getent passwd john
john:x:10000:5000:John Doe:/home/john:/bin/bash

The same test on arm64 or ppc64el platform where the libnss-ldap have been rebuilt recently you will get :

root@trusty-ldapclient:~# getent passwd john
Segmentation fault (core dumped)

$ sudo apt-get download libnss-ldap
$ mkdir tmp
$ dpkg -x libnss-ldap_264-2.2ubuntu4_amd64.deb tmp
$ nm -D tmp/lib/x86_64-linux-gnu/libnss_ldap-2.15.so |grep lock$
                 w __pthread_mutex_lock
                 w __pthread_mutex_unlock
#Rebuild the library
$ pull-lp-source libnss-ldap trusty
$ sbuild -A -d trusty libnss-ldap_264-2.2ubuntu4.dsc
$ rm -Rf tmp/*
$ dpkg -x libnss-ldap_264-2.2ubuntu4_amd64.deb tmp
$ nm -D tmp/lib/x86_64-linux-gnu/libnss_ldap-2.19.so | grep lock$
                 U __libc_lock_lock
                 U __libc_lock_unlock

Notice that the libnss-ldap version change (2.15 -> 2.19). With the newly built version, the expected _pthread_mutex_lock|unlock is no longer present.

[Regression]
None expected. This is already present and in use in the upstream version of the library.

15:27 <rbasak> caribou: so what I don't like about this is that the patch seems a bit invasive in an area where if there's a regression, it'll be in multithreading code that'll be non-deterministic and thus difficult to test.

15:27 <rbasak> caribou: OTOH, it's broken on ppc64el at the moment? That means we need to fix it.

15:28 <rbasak> Having an active upstream that had committed the code would give me more confidence that the patch is good (since they're more familiar with the code and will have reviewed it)

15:29 <rbasak> But Debian have committed it, so that's better than nothing.

15:31 <rbasak> caribou: I think we have no choice but to push it to Trusty (and Utopic), but we should let the SRU team decide at that stage. IMHO, my concern should be noted in "Regression Potential", so I'll do that now.

[Original Description of the problem]

Unlike previously thought, this bug is _NOT_ specific to the PPC64EL architecture. More details in the analysis.

Many commands that require the use of libnss-ldap will fail with Segmentation Fault. The boot procedure itself can be blocked with the following message.

One potential workaround is to remove the use of ldap from the /etc/nsswitch.conf file to at least provide a bootable system.

Louis Bouchard (louis)
Changed in libnss-ldap (Ubuntu):
status: New → Confirmed
importance: Undecided → Critical
Changed in libnss-ldap (Ubuntu Utopic):
status: New → Confirmed
importance: Undecided → Critical
description: updated
Revision history for this message
Louis Bouchard (louis) wrote :

Analysis
========

Upon boot of a ppc64el system configured to authenticate as an LDAP client, the boot process is interrupted with :

 /sbin/init: symbol lookup error: /lib/powerpc64le-linux-gnu/libnss_ldap.so.2: undefined symbol: __libc_lock_lock

Apparently __libc_lock_lock is missing from the libnss_ldap.so library.

Looking at the symbols exported by the library on PPC64el we see :

 - For libnss_ldap :
$ nm -D /lib/powerpc64le-linux-gnu/libnss_ldap.so.2 | grep _lock
                 U __libc_lock_lock
                 U __libc_lock_unlock

• For libc :
$ nm -D /lib/powerpc64le-linux-gnu/libc-2.19.so | grep _lock
000000000008f4a0 T _IO_list_lock
0000000000088a70 T _IO_peekc_locked
00000000001313b0 T pthread_mutex_lock

The same command on AMD64 returns :
 - For libnss_ldap:
# nm -D /lib/x86_64-linux-gnu/libnss_ldap.so.2 | grep lock
                 w __pthread_mutex_lock
                 w __pthread_mutex_unlock
• For libc :
# nm -D /lib/x86_64-linux-gnu/libc-2.19.so | grep _lock
...
0000000000108270 T pthread_mutex_lock

So the symbol is _not_ exported by the libc on ppc64el even though libnss_ldap is expecting to find it.

This lock is only used once in libnss_ldap :

static int
ltf_mutex_lock (void *mutexp)
{
#if defined(HAVE_LIBC_LOCK_H) || defined(HAVE_BITS_LIBC_LOCK_H)
  return __libc_lock_lock (*(pthread_mutex_t *) mutexp);
#elif defined(HPUX)
  return __thread_mutex_lock ((pthread_mutex_t *) mutexp);
#else
# ifdef _AIX
  if (__multi_threaded == 0)
    return 0;
# endif
  return pthread_mutex_lock ((pthread_mutex_t *) mutexp);
#endif /* HAVE_LIBC_LOCK_H || HAVE_BITS_LIBC_LOCK_H */
}

So apparently, the HAVE_LIBC_LOCK_H symbol is defined on ppc64el, or at least appear to be even if the libc does not export that libc_lock_lock symbol.

Louis Bouchard (louis)
Changed in libnss-ldap (Ubuntu):
assignee: nobody → Louis Bouchard (louis-bouchard)
Changed in libnss-ldap (Ubuntu Utopic):
assignee: nobody → Louis Bouchard (louis-bouchard)
Revision history for this message
Louis Bouchard (louis) wrote :

Ok, here are some more puzzling things :

The libnss-ldap from the archive does not use __libc_lock_lock :
$ pwd
./archive
$ wget http://fr.archive.ubuntu.com/ubuntu/pool/main/libn/libnss-ldap/libnss-ldap_264-2.2ubuntu4_amd64.deb
--2014-10-30 16:20:59-- http://fr.archive.ubuntu.com/ubuntu/pool/main/libn/libnss-ldap/libnss-ldap_264-2.2ubuntu4_amd64.deb
...
2014-10-30 16:21:01 (72,7 KB/s) - «libnss-ldap_264-2.2ubuntu4_amd64.deb» enregistré [64718/64718]
$ dpkg -x libnss-ldap_264-2.2ubuntu4_amd64.deb .
$ nm -D lib/x86_64-linux-gnu/libnss_ldap.so.2 | grep _lock
                 w __pthread_mutex_lock

The same source build in a PPA DOES have the symbol __libc_lock_lock :
$ cd ../ppa
$ pwd
./ppa
$ wget https://launchpad.net/~louis-bouchard/+archive/ubuntu/nssldap-buildtest/+files/libnss-ldap_264-2.2ubuntu4_amd64.deb
...
2014-10-30 16:23:56 (56,7 KB/s) - «libnss-ldap_264-2.2ubuntu4_amd64.deb» enregistré [83234/83234]

$ dpkg -x libnss-ldap_264-2.2ubuntu4_amd64.deb .
$ nm -D lib/x86_64-linux-gnu/libnss_ldap.so.2 | grep _lock
                 U __libc_lock_lock
                 U __libc_lock_unlock

I see the same behavior (i.e. fresh build created with __libc_lock_lock in use) when using sbuild or building manually using debian/rules

This needs more investigation

Revision history for this message
Louis Bouchard (louis) wrote :

After a discussion with slangasek, the only reason why it no longer builds locally is that the uploads in the archives have not been rebuilt since 2012 which explains why the new build on ppc64el fails : it is the only fresh rebuild.

Changed in libnss-ldap (Ubuntu Trusty):
status: New → Confirmed
Changed in libnss-ldap (Ubuntu Precise):
importance: Undecided → Critical
Changed in libnss-ldap (Ubuntu Trusty):
importance: Undecided → Critical
Changed in libnss-ldap (Ubuntu Precise):
assignee: nobody → Louis Bouchard (louis-bouchard)
Changed in libnss-ldap (Ubuntu Trusty):
assignee: nobody → Louis Bouchard (louis-bouchard)
Revision history for this message
Louis Bouchard (louis) wrote :

More research have highlighted what seems like a potential solution :

 * https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=727177
                  "Upgrade of libnss-ldap to 265-1 causes important binaries to segfault"

The debian bug description is in line with what is described above. The version 265-3 should be merged into Vivid, then the specific patch for this problem will be SRUed into the impacted releases.

Louis Bouchard (louis)
Changed in libnss-ldap (Ubuntu):
status: Confirmed → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libnss-ldap - 265-3ubuntu2

---------------
libnss-ldap (265-3ubuntu2) vivid; urgency=low

  * Changelog entry enhanced from previous version with more details.
  * debian/changelog : removed po-debconf from Build-depends
  * Merge from Debian unstable. Remaining changes:
    (LP: #1389152)
    + debian/control: update Maintainer field
      - Removed cdbs, quilt, patchutils as the build now uses dh-autoreconf
      - Kept po-debconf out of Build-depends
      - Move nscd to Suggests
    + removed the following files:
      - debian/config
      - debian/libnss-ldap.postinst
      - debian/libnss-ldap.postrm
      - debian/po
      - debian/templates
    + modified the following files:
      - debian/libnss-ldap.dirs - Adapted to new build structure
      - debian/libnss-ldap.install - Adapted to new build structure
    + debian/LDAP-Permissions.txt: use /etc/ldap.conf
    + debian/README.Debian: use /etc/ldap.conf
    + debian/rules: don't use --with-ldap-conf-file and --with-ldap-secret-file
    + debian/libnss-ldap.install: removed usr/share/libnss-ldap/ldap.conf
    + debian/libnss-ldap.dirs: removed usr/share/libnss-ldap
    + debian/patches :
      - authpriv_log_c & authpriv_log_h merged into log_authpriv.patch
      - autoreconf.patch dropped as dh-autoreconf is used now
      - debian_conf is now conf_notes.patch
      - fix-ethers-truncation.patch is dropped since fixed upstream
      - fix-glibc-test-for-armel-gnueabi.patch replaced by
        treat-all-debian-systems-like-linux.patch
      - glibc-2.16.patch Handle removal of __libc_lock_lock and similar
      symbols from libc (LP: #1387594)
  * Remaining changes from debian unstable for LP: #155947
    + added debian/nssldap-update-ignoreusers which update
      nss_initgroups_ignoreusers in /etc/ldap.conf based on
      nss_initgroups_minimum_uid
    + add manpage for nssldap-update-ignoreusers
    + debian/libnss-ldap.install: added usr/sbin/nssldap-update-ignoreusers
      and usr/share/man/man8/nssldap-update-ignoreusers.8
    + added debian/libnss-ldap.init to call nssldap-update-ignoreusers on
      shutdown
    + debian/rules: install initscript
    + debian/libnss-ldap.dirs: added var/lib/libnss-ldap
    + debian/patches/minimum_uid.patch: update ldap.conf and manpage for
      nss_initgroups_ignoreusers and nss_initgroups_minimum_uid updates;
 -- Louis Bouchard <email address hidden> Mon, 03 Nov 2014 17:03:35 +0100

Changed in libnss-ldap (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Louis Bouchard (louis) wrote :

The version in Precise is unaffected. Marking the task invalid

Changed in libnss-ldap (Ubuntu Precise):
status: New → Invalid
Changed in libnss-ldap (Ubuntu Trusty):
status: Confirmed → In Progress
Changed in libnss-ldap (Ubuntu Utopic):
status: Confirmed → In Progress
Changed in libnss-ldap (Ubuntu):
assignee: Louis Bouchard (louis-bouchard) → nobody
tags: added: cts
Revision history for this message
Louis Bouchard (louis) wrote :

debdiff for the SRU on Trusty

description: updated
Revision history for this message
Louis Bouchard (louis) wrote :

debdiff for SRU on Utopic

Changed in libnss-ldap (Ubuntu Precise):
assignee: Louis Bouchard (louis-bouchard) → nobody
Louis Bouchard (louis)
description: updated
Revision history for this message
Matt Ezell (ezellma) wrote :

Anything I can do to help this SRU along? This is affecting our Trusty and Utopic boxes. Right now, I've manually pulled in a Vivid package so I actually have working systems. It would be nice to get this into the *-updates repos.

Revision history for this message
Louis Bouchard (louis) wrote :

looks like I forgot to attach the trusty debdiff

Robie Basak (racb)
description: updated
Revision history for this message
Robie Basak (racb) wrote :

Uploaded to Trusty and Utopic with minor changes as discussed with Louis on IRC.

Note that I've looked at the patches, but they're too in depth for me to follow really. Superficially they look fine, and Debian have also included them so they're in Vivid, and so I think this is sufficient provided we do the usual SRU testing. Note my thoughts about regression potential in the bug description, though.

Thanks Louis!

Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Louis, or anyone else affected,

Accepted libnss-ldap into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/libnss-ldap/264-2.2ubuntu4.14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in libnss-ldap (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in libnss-ldap (Ubuntu Utopic):
status: In Progress → Fix Committed
Revision history for this message
Adam Conrad (adconrad) wrote :

Hello Louis, or anyone else affected,

Accepted libnss-ldap into utopic-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/libnss-ldap/264-2.2ubuntu4.14.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Matt Ezell (ezellma) wrote :

I ran tests on an IBM 8247-22L using both Utopic and Trusty for ppc64el. Details of the test follow, but the new packages in [utopic,trusty]-proposed fix the segfault issue.

#### For Utopic ####
root@crest2:~# lsb_release -rc
Release: 14.10
Codename: utopic
# Current working version using a vivid package
root@crest2:~# dpkg -l libnss-ldap|grep ii
ii libnss-ldap:ppc64el 265-3ubuntu2 ppc64el NSS module for using LDAP as a naming service
root@crest2:~# getent passwd|wc -l ; getent group|wc -l
52
16849
# Test current release
root@crest2:~# apt-get install libnss-ldap=264-2.2ubuntu4
...
root@crest2:~# getent passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
... other local users ...
Segmentation fault
root@crest2:~# getent group
root:x:0:
daemon:x:1:
... other local groups ...
Segmentation fault
# Try new version
root@crest2:~# apt-get install libnss-ldap=264-2.2ubuntu4.14.10.1
...
root@crest2:~# getent passwd|wc -l ; getent group|wc -l
52
16849
# Success!

#### For Trusty ####
root@crest2:~# lsb_release -rcd
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty
# Test current release
root@crest2:~# apt-get install libnss-ldap=264-2.2ubuntu4
...
root@crest2:~# getent passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
... other local users ...
Segmentation fault
root@crest2:~# getent group
root:x:0:
daemon:x:1:
... other local groups ...
Segmentation fault
# Try new version
root@crest2:~# apt-get install libnss-ldap=264-2.2ubuntu4.14.04.1
...
root@crest2:~# getent passwd|wc -l ; getent group|wc -l
52
16849
# Success!

tags: added: verification-done
removed: verification-needed
Revision history for this message
Brad Blondin (brad-blondin-p) wrote :

Thanks, the patch libnss-ldap=264-2.2ubuntu4.14.04.1 works for me

Revision history for this message
Matt Ezell (ezellma) wrote :

Hi Adam- Is this SRU appropriate to target for the 14.04.2 milestone?

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libnss-ldap - 264-2.2ubuntu4.14.04.1

---------------
libnss-ldap (264-2.2ubuntu4.14.04.1) trusty; urgency=medium

  * Handle removal of __libc_lock_lock and similar symbols from libc
    (LP: #1387594). This fixes a segfault on ppc64el, and also all other
    architectures in case of a future rebuild.
 -- Louis Bouchard <email address hidden> Wed, 12 Nov 2014 16:59:29 +0100

Changed in libnss-ldap (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for libnss-ldap has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libnss-ldap - 264-2.2ubuntu4.14.10.1

---------------
libnss-ldap (264-2.2ubuntu4.14.10.1) utopic; urgency=medium

  * Handle removal of __libc_lock_lock and similar symbols from libc
    (LP: #1387594). This fixes a segfault on ppc64el, and also all other
    architectures in case of a future rebuild.
 -- Louis Bouchard <email address hidden> Wed, 12 Nov 2014 16:29:42 +0100

Changed in libnss-ldap (Ubuntu Utopic):
status: Fix Committed → Fix Released
Changed in libnss-ldap (Debian):
status: Unknown → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.