nscd leaking memory using libnss-ldap
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| eglibc (Ubuntu) |
Fix Released
|
High
|
Mathias Gug | ||
| Hardy |
Won't Fix
|
High
|
Unassigned | ||
| Intrepid |
Won't Fix
|
High
|
Unassigned | ||
| Jaunty |
Won't Fix
|
High
|
Unassigned | ||
| Karmic |
Won't Fix
|
High
|
Unassigned | ||
| Lucid |
Fix Released
|
High
|
Mathias Gug | ||
Bug Description
We've got our ubuntu servers (8.04.1 LTS) authenticating users against an active directory libpam-krb5 and user account info is gotten via libnss-ldap and nscd. However, the nscd process keeps growing in memory image size until it fills the system memory completely. Memory leak speed seems to correlate to the number of queries against nss.
The AD is Windows 2003 R2, using the R2 provided Identity management for UNIX (ex-SFU) to provide LDAP attributes. There are about 50 user objects and a dozen group objects matching the search filters specified in /etc/ldap.conf, plus the stuff that goes with AD by default.
On our mail server, which uses nss queries the most and suffers worst from this problem, the memory usage gets up to 95MB (RES) and 203MB (VIRT) in 12 hours as observed by top, and exceeds one gigabyte in 3-5 days.
I have tried turning paranoia mode on to restart the service periodically, but for some reason it does not restart, just quits the whole process (I don't know whether the paranoia mode is actually supposed to workin Ubuntu, as it is not mentioned in documentation provided with Ubuntu release, but it's parameters are in default config file, so I decided to try).
Just tried to install a fresh Ubuntu 8.04.1 server where I installed only libnss-ldap, nscd and their requirements, configured it for AD connection and left idle for the night. nscd memory usage was up to 100MB in the morning when it initially (few minutes after restart) is about 3.5MB.
*** Release and package info ***
Description: Ubuntu 8.04.1
Release: 8.04
nscd:
Installed: 2.7-10ubuntu4
Candidate: 2.7-10ubuntu4
Version table:
*** 2.7-10ubuntu4 0
500 http://
100 /var/lib/
2.7-10ubuntu3 0
500 http://
*** Sanitized /etc/ldap.conf ***
base dc=our,
uri ldaps:/
ldap_version 3
rootbinddn <email address hidden>
scope sub
pam_password crypt
nss_base_passwd dc=our,
nss_base_group dc=our,
nss_map_objectclass posixAccount user
nss_map_objectclass shadowAccount user
nss_map_attribute uid sAMAccountName
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_attribute shadowLastChange pwdLastSet
nss_map_objectclass posixGroup group
nss_map_attribute uniqueMember member
pam_login_attribute sAMAccountName
pam_filter objectclass=User
nss_map_attribute userPassword unixUserPassword
ssl on
tls_checkpeer no
sasl_secprops maxssf=0
nss_initgroups_
*** Password for <email address hidden> in /etc/ldap.secret ***
*** Using default package provided /etc/nscd.conf ***
Related branches
| Doug Engert (deengert) wrote | #1 |
| Doug Engert (deengert) wrote | #2 |
| Doug Engert (deengert) wrote | #3 |
| affects: | glibc (Ubuntu) → libnss-ldap (Ubuntu) |
| Doug Engert (deengert) wrote | #4 |
| Changed in libnss-ldap (Ubuntu Lucid): | |
| milestone: | none → ubuntu-10.04-beta-2 |
| status: | New → Triaged |
| importance: | Undecided → High |
| Changed in libnss-ldap (Ubuntu Hardy): | |
| status: | New → Triaged |
| importance: | Undecided → High |
| Changed in libnss-ldap (Ubuntu Intrepid): | |
| status: | New → Triaged |
| importance: | Undecided → High |
| Changed in libnss-ldap (Ubuntu Jaunty): | |
| status: | New → Triaged |
| importance: | Undecided → High |
| Changed in libnss-ldap (Ubuntu Karmic): | |
| status: | New → Triaged |
| importance: | Undecided → High |
| Changed in libnss-ldap (Ubuntu Lucid): | |
| assignee: | nobody → Mathias Gug (mathiaz) |
| Changed in libnss-ldap (Ubuntu Lucid): | |
| status: | Triaged → In Progress |
| Changed in libnss-ldap (Ubuntu Lucid): | |
| status: | In Progress → Fix Committed |
| Mathias Gug (mathiaz) wrote | #5 |
| Launchpad Janitor (janitor) wrote | #6 |
| Changed in libnss-ldap (Ubuntu Lucid): | |
| status: | Fix Committed → Fix Released |
| Chuck Short (zulcss) wrote | #7 |
| Changed in libnss-ldap (Ubuntu Intrepid): | |
| status: | Triaged → Won't Fix |
| affects: | libnss-ldap (Ubuntu Lucid) → eglibc (Ubuntu Lucid) |
| Changed in eglibc (Ubuntu Jaunty): | |
| status: | Triaged → Won't Fix |
| Changed in eglibc (Ubuntu Karmic): | |
| status: | Triaged → Won't Fix |
| Rolf Leggewie (r0lf) wrote | #8 |
| Changed in eglibc (Ubuntu Hardy): | |
| status: | Triaged → Won't Fix |
A memory leak in libnss-ldap over time can cause the nscd
process to grow extremely large. For example one nscd
process that had been running for three months was using
6GB of swap!
The problem is in the original Padl nss-ldap in at least versions
258, 261 and 265. Ubuntu Hardy uses 258, Karmic uses 261, and
the Padl current release is 265.
The ldap-nss.c do_init() may be called more then once,
to initialize an ldap session and save the session in
in __session.ls_conn and set the __session.ls_stat = LD_INITIALIZED
But it does not check the state to see if has be initialized,
and at line 1239: __session.ls_conn = NULL;
The attached patch to to the libnss-
the problem, by testing __session.ls_stat == LD_INITIALIZED
The ldap-nss.c also has a patch to call do_close twice that
I had previously turned it to Padl and is now in 265.
For testing purposes, the patch also adds atexit(do_atexit);
and a do_atexit routine to call do_close. This will then cause
the last session to be be freed. This make it much easier to
use valgrind to check for memory leaks. (in the nscd.c in lib6c
the _exit was change to exit so the atexit would be called.)
Debug versions of nscd and libnss-ldap where created, and
and used with valgrind to track down the memory leaks.
Attached is a script used with valgrind. The LD_PRELOAD was
needed so dynamic libs would not be unloaded, and valgrind
could find the symbol tables.