slapd package configuration aborts with "Program version 4.7 doesn't match environment version 0.44" error during Hardy -> Lucid upgrade

This issue seems to be caused by the fact that the openldap packages switched from using v4.2 to v4.7 of the Berkeley Database libraries, as mentioned in the changelog.Debian.gz file:
   openldap (2.4.14-0ubuntu1) jaunty; urgency=low
   [...]
     * Build against db4.7 instead of db4.2 at last! Closes: #421946.
   [...]
    -- Mathias Gug <email address hidden> Wed, 18 Feb 2009 18:44:00 -0500

(Since this happened during the Jaunty timeframe, it would make sense that an upgrade from Hardy to Lucid would be affected.)

The best discussion about changing to a new BDB version that I found was an old README.db4.4 that was part of the Debian "subversion" package back in 2006. The document has since been removed from that package, once Subversion started handling the DB conversion automatically, but the text is still available from the source repository for the subversion package:
  http://svn.debian.org/wsvn/pkg-subversion/src/tags/1.4.0-1/debian/README.db4.4

The OpenLDAP changelog files and other discussions out on the web imply that slapd normally does automatic "recovery" of the BDB data files, which I understand should transparently take care of situations where the library version has changed.

However, in the case of the Ubuntu package upgrade, the "slaptest" command is getting executed before the new version of "slapd" is run, and (given the error that is occuring) it would appear that "slaptest" does not performat that same automatic recovery operation....

More specifically, the slapd.postinst script is attempting to use the "slaptest" command to migrate the slapd.conf config file to new slapd.d format, but in the case where old-BDB-version database files exist under /var/lib/ldap, that command errors out, thus causing the postinst script to abort.

In my case, I was able to get past this situation by using the commands described in the above-mentioned README.db4.4 file, as follows:
   (first, install the db4.2-util package)
   # cd /var/lib/ldap
   # db4.2_checkpoint -1
   # db4.2_recover
   # db4.2_archive -d

After that I was able to run a plain "slaptest" command manually without getting the "Program version" error, and when I did another
    # dpkg --pending --configure
, the slapd.postinst script's attempt to convert the configuration info was finally successful.

As far as I know there was nothing unusual about my slapd instance and associated data files before the upgrade -- but I don't have any other installation to compare them to, so it's possible that not all Hardy -> Lucid upgrades will trigger this issue. Still, I'm wondering if there is any way that the slapd.postinst script could detected and avoid this situation without requiring manual intervention. (And if not, whether this issue should be mentioned on the Lucid release notes?)

I did some more investigation into this issue, and it looks like this bug will affect all Hardy -> Lucid upgrades.

To test this I went through the following steps:

On a machine running Hardy (and which had never had "slapd" installed before):
1a) installed the "slapd" package, allowed the postinst script to configure the installation in the normal manner, and then ran "/etc/init.d/slapd stop" to cleanly shut down the daemon process.

1b) ran "slaptest -f /etc/ldap/slapd.conf" and confirmed that the BDB v4.2 database files did not cause any errors on that machine.

Back on the server recently upgraded to Lucid, I made a simple test directory:
2a) Created a test directory "/root/ldap_hardy_test", and under that an empty subdirectory "libldap".

2b) copied /etc/ldap/slapd.conf.old to ldap_hardy_test/slapd.conf, and edited the "directory" line to point to the /root/ldap_hardy_test/libldap directory.

2c) copied all files in /var/lib/ldap/ into /root/ldap_hardy_test/libldap/

2d) ran "slaptest -f slapd.conf -d 1" to confirm that the slaptest run completed without errors, and that it opened the files in ldap_hardy_test/libldap instead of /var/lib/ldap .

Then I tested the BDB files from the Hardy install:
3a) deleted all files in /root/ldap_hardy_test/libldap, and then copied all the files from the Hardy machine's /var/lib/ldap/ directory into the Lucid machine's "libldap" directory.

3b) ran "slaptest -f slapd.conf" again... and got the " Program version 4.7 doesn't match environment version" error message.

So, it appears that the BDB database files from even the simplest Hardy slapd install with trigger this error....

Mathias, can you take a look at this for beta 2?

(As a side note, I noticed that the actual version number printed out in the "doesn't match environment version 0.XXX" message was not consistent -- when I did the original upgrade it was "0.44", but in my later testing it started out at 0.143 and then kept getting larger.

It turns out that the format of the Berkeley DB "environment" file changed between BDB 4.3 and 4.4, and in particular the location of the bytes that encode the database version number changed. In the the case of lucid's OpenLDAP tools (which use DBD v4.7) looking at the environment file left over from hardy (v4.2), the bytes that are interpreted as the "minor" version number actually contains the "Locks granted without waiting" count.

So, for example, if I run
  db4.2_stat -e -N | head | grep "without waiting"
from within the BDB directory, the count shown will match the XXX printed in the "environment version 0.XXX" message from slaptest.

This means that running some other command that changes the locks-granted count [e.g. "db4.2_stat -e" , without the "-N" ] will actually cause the XXX number to change between different attempts to run slaptest .

It seems like the header format of the environment file was stabilized as of db4.4, so db4.7 tools shouldn't have the same problem with files generated by 4.4 and later. )

I took a closer look at the slapd.postinst script, and I believe I see what is causing this issue.

In the "postinst_upgrade_configuration" function, the script first checks to see if the configuration info needs to be converted from "slapd.conf" to "slapd.d" format, and if so it runs the "slaptest" command to perform that conversion.

The script then checks to see if the previous version of the of package used a different Berkeley DB version, and if so it actually moves the old BDB files out of the way and creates new database files using the "slapadd" program (based on the export of the directory data that was saved using the prior version of "slapcat" in the slapd.preinst script).

This arrangement worked fine for the Hardy -> Intrepid -> Jaunty upgrade path, because the two conversions did not happen during the same run of the postinst script. (The configuration was converted when going to Intrepid, and the BDB version was changed during the upgrade to Jaunty.)

However, the direct Hardy -> Lucid upgrade does need to perform both conversions in the same postinst run -- but in the existing arrangement, the script tries to run "slapadd" before the BDB files have been regenerated, and thus triggering the "Program version 4.7 doesn't match environment version" error....

So, it appears that the solution to this problem is to perform the conversions in the opposite order.

Currently, the "load_databases" function assumes that the configuration information is in the directory format, so it can't be used until after that conversion has happened. However, it seems like it should be fairly easy to tweak it to work with either configuration format (e.g. using the same logic as is currently found in the "dump_databases" function). The two conversion steps in "postinst_upgrade_configuration" could then be switched, which would (hopefully) allow both to be performed successfully during the same upgrade....

This bug was fixed in the package openldap - 2.4.21-0ubuntu3

---------------
openldap (2.4.21-0ubuntu3) lucid; urgency=low

  * debian/slapd.postinst, debian/slapd.scripts-common: Upgrade databases
    before trying to convert to slapd.d, to avoid upgrade failure from hardy
    (LP: #536958)
  * debian/slapd.postinst: Add a {1} numeric index to olcAccess entry in
    olcDatabase={0}config.ldif to avoid upgrade failures (LP: #538516, #526230)
 -- Thierry Carrez <email address hidden> Mon, 29 Mar 2010 13:31:47 +0200

I just did another hardy -> lucid upgrade run (on a test machine running an as-installed-by-the-package slapd configuration), and can confirm that the new version of the slapd.postinst was able to complete without triggering the "Program version 4.7 doesn't match environment version" error.