Crashes with undefined symbol

Status changed to 'Confirmed' because the bug affects multiple users.

Produces the following error in the dovecot logs:

Jan 01 15:58:11 imap(toaster): Error: Couldn't load required plugin /usr/lib/dovecot/modules/lib21_fts_lucene_plugin.so: dlopen() failed: /usr/lib/dovecot/modules/lib21_fts_lucene_plugin.so: undefined symbol: _ZN6lucene8analysis8snowball16SnowballAnalyzerC1EPFiPKvmP6bufferEPKc

This bug was fixed in the package dovecot - 1:2.2.22-1ubuntu4

---------------
dovecot (1:2.2.22-1ubuntu4) xenial; urgency=medium

  * Urgh. Previous upload added a runtime dependency on libstemmer0d to
    dovecot-core that I missed. Revert it, and reopen LP: 1524526.

 -- Christopher James Halse Rogers <email address hidden> Thu, 21 Apr 2016 09:59:28 +1000

1:2.2.22-1ubuntu4 does not appear to be available from the current Ubuntu repos. I have updated today and can only get 1:2.2.22-1ubuntu2

I see that 1:2.2.22-1ubuntu4 is available in Yakkety, I have pulled, compiled and installed it.

apt show dovecot-core
Package: dovecot-core
Version: 1:2.2.22-1ubuntu4
Status: install ok installed

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04 LTS"

This happens:

doveadm pw -s SSHA256 -p verysecretindeed
Fatal: Couldn't load required plugin /usr/lib/dovecot/modules/lib21_fts_lucene_plugin.so: dlopen() failed: /usr/lib/dovecot/modules/lib21_fts_lucene_plugin.so: undefined symbol: _ZN6lucene8analysis8snowball16SnowballAnalyzerC1EPFiPKvmP6bufferEPKc

So, on my machine, this bug is NOT fixed.

aptitude search dovecot | grep -P '^i'
i A dovecot-core - secure POP3/IMAP server - core files
i dovecot-imapd - secure POP3/IMAP server - IMAP daemon
i dovecot-lmtpd - secure POP3/IMAP server - LMTP server
i dovecot-lucene - secure POP3/IMAP server - Lucene support
i dovecot-managesieved - secure POP3/IMAP server - ManageSieve serv
i dovecot-sieve - secure POP3/IMAP server - Sieve filters su
i dovecot-sqlite - secure POP3/IMAP server - SQLite support

Yes; 22-1ubuntu3 fixed it, but introduced an universe dependency on a main package. This was referred and uploaded as 22-1ubuntu4, which is why the changelog entry mentions re-opening this bug.

Thank you for the explanation. I had failed to understand the issue.
However, that does leave us with completely broken version of Dovecot
because doveadm is something that is used all over the place.

On 07/05/16 23:20, Chris Halse Rogers wrote:
> Yes; 22-1ubuntu3 fixed it, but introduced an universe dependency on a
> main package. This was referred and uploaded as 22-1ubuntu4, which is
> why the changelog entry mentions re-opening this bug.
>

It should only crash if you have dovecot-lucene installed and enabled. If that's not the case, your running into a different bug.

Since it was only 98% clear :-), here a summary of a short and easy guide to trigger the issue in e.g. a container:

#1 install dovecot+lucene
sudo apt-get install dovecot-lucene
#2 configure lucene
sed -ie 's/#mail_plugins = $/mail_plugins = $mail_plugins fts fts_lucene/' /etc/dovecot/conf.d/10-mail.conf
sed -ie 's/^plugin {$/plugin {\n fts = lucene\n fts_lucene = whitespace_chars=@./' /etc/dovecot/conf.d/90-plugin.conf
#3 do soething with dovecot
service dovecot reload
doveadm pw -s SSHA256 -p verysecretindeed

Fatal: Couldn't load required plugin /usr/lib/dovecot/modules/lib21_fts_lucene_plugin.so: dlopen() failed: /usr/lib/dovecot/modules/lib21_fts_lucene_plugin.so: undefined symbol: _ZN6lucene8analysis8snowball16SnowballAnalyzerC1EPFiPKvmP6bufferEPKc

Note that while it won't save us from this being a bug, the comment at http://wiki2.dovecot.org/Plugins/FTS/Lucene is worth to note as a lucene user:
Quote: "NOTE: Although the fts-lucene plugin works, it's using CLucene library, which is very old and has some bugs. It's a much better idea to use fts-solr instead, which has much more features and is more stable."

Initial Analysis:
- stemmer can be set at the configure step, but it isn't by d/rules (default is auto)
- providing libstemmer-dev converts the auto to a yes (as happened in the ubuntu3 version)
  - that yes then builds the code that is missing regarding this error here
  - but also builds the stemmer dependency into dovecot-core (main)
- textcat is also not given by rules and defaults to auto (yes, due to exttectcat being a build dependency)
- plugin config "no_snowball" is no workaround as it fails already at ld.so

Checking Sources and Build:
- Build logs should be free of Snowball, but aren't
   => even without libstemmer I still see snowball reference
- BUILD_FTS_STEMMER guards inclusion of Snowball.cc in fts-lucene/Makefile.am
- But due to complexities in automake BUILD_FTS_EXTTEXTCAT_TRUE guards Snowball.cc in fts-lucene/Makefile.in eventually

I wondered if this is a Regression at all?:
- <=vivid had dovecot 2.2.9 which didn't have the lucene plugin
- >=wily had lucene in the currently failing form

Options for us:
1. - drop textcat dev from build deps, but that will also remove other functionality e.g. lib-fts/fts-language.c
 => No, we don't want to cut other parts of dovecot
2. - get libstemmer into main to make the runtime dependency a no-issue
 => No, not reasonable for this alone
3. - try to guard the critical code via libstemmer instead of libtextcat (possible but complex for us not owning/used to that code)
 => Maybe (could be done, but likely error prone)
4. - drop lucene plugin
 => This would be my choice, especially since:
  - lucene is deprecated in favor of solr by the dovecot project
  - and it never worked since it's appearance in Ubuntu

Obviously this is up for discussion ...

FYI about Debian on that situation.
- They are on 2.2.24 (so a merge is outstanding), but that is not affecting this situation.
- Debian has the libstemmer-dev build dependency since 2.2.12-2 (Mar 2014)
- Our merges usually have (as the last by Doko) "Drop build dependency on libstemmer-dev (universe)."

So either now or on the next merge we should/could instead of just:
  "Drop build dependency on libstemmer-dev (universe)."
Do
  - Disable plugin dovecot-lucene as it is deprecated in favor of solr.
  - Drop build dependency on libstemmer-dev (universe) only needed for dovecot-lucene.

Raof, Racb and I just agreed on removing the dovecot-lucene.

Steps to be taken:
#1
- Merge to .24 in the Dev release (Debian is ahead atm)
- on that merge drop dovecot-lucene package
- should be case 11 on https://wiki.debian.org/PackageTransition
- discuss with if that makes sure it gets removed on Upgrade (on the prepare example)

#2
- After in accepted in Dev Release, care about a bit special "SRU" then
- While not possible to remove old packages, there seems to be a way to make an empty package
- Discuss with SRU Team if/how that can be done

Short FYI
I have prepared a merge including the change for #1.

Testing upgrades with minimal as well as full dovecot installs work well now.
A system with fts_lucene plugin set up for dovecot will still complain, but only about the plugin not being there which is correct as we removed it.
The config can be done in too many ways, so it can hardly be auto-fixed and it probably would be soo much of a hammer (for something that never worked) to pull in debconf for prompting just for that.

Still the issue will be fixed, and other than those few who set it up with the issues it had before no one should run into any trouble.

While testing all of that I found that the ADT test seems to be broken (even before any of my changes or my merge), so I have at least to look into that before proposing the merge.

dovecot-antispam will need a no-change-rebuild once the merge for dovecot is complete.
Adding as affected and uploading the debdiff for that now.

Merge prepared for review in https://code.launchpad.net/~paelzer/ubuntu/+source/dovecot/+git/dovecot/+merge/298040

The attachment "no change rebuild for new dovecot abi" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Please clean this up so that sponsors actually know what to do. My best guess:

 - the merge apply to yakkety
 - the "rebuild only" patch (which says "yakkety") is targetted at xenial
 - the dovecot tasks shoudl be invalid

?

Hi Martin,
no need to guess - and sorry it wasn't meant to be sponsored yet.
Sponsors became auto-subscribed by the Bot due to the debdiff I prepared.

But all of that only becomes ready and meaningful once the dovecot merge is completed (which currently is in review).

I can't by LP permissions, but please feel free to unsubscribe sponsors for now to avoid further confusion. I'll add you again once all the dependencies are ready.

Thanks for the follow-up; unsubscribed sponsors for now then, please re-subscribe once this is ready.

This bug was fixed in the package dovecot - 1:2.2.24-1ubuntu1

---------------
dovecot (1:2.2.24-1ubuntu1) yakkety; urgency=medium

  * Merge with Debian; Remaining Changes:
    + Add autopkgtest to debian/tests/*.
    + Drop build dependency on libstemmer-dev (universe)
    + Use Snakeoil SSL certificates by default
      - d/control: Depend on ssl-cert
    + add lsb base dependency to ensure debian/dovecot-core.dovecot.init is
      working correctly
    + Add ufw integration:
      - d/dovecot-core.ufw.profile: new ufw profile.
      - d/rules: install profile in dovecot-core.
      - d/control: dovecot-core - suggest ufw.
    + Add apport hook:
      - d/rules, d/source_dovecot.py
    + Remove lintian override for drac
    + Add mail-stack-delivery
      - add package in d/rules, d/control
      - add d/*mail-stack-delivery* maintainer scripts and default conf
      - d/mail-stack-delivery.preinst: Move previously installed backups and
        config files to a new package namespace.
  * Added Changes:
    - Disable dovecot-lucene plugin as it had various issues, has universe
      dependencies and is deprecated in favor of solr anyway (LP: #1524526).
    - Fixup the autopkgtests we add to work with recent dovecot versions
    - d/mail-stack-delivery.README.Debian clarified use of configuration files
    - handle conffile removal of /etc/init/dovecot.conf (due to dropping
      upstart). Can be removed once no upgrade path from <yakkety is left.
  * Dropped Changes:
    - Add upstart job (that means we no more add it now)
    - no more needed upgrade handling of mail-stack-delivery related to
      2.1.7-7 and 1.2.9-1ubuntu8 (both out of scope of any possible
      paths now)
    - d/dovecot-core.dirs: Added usr/share/doc/dovecot-core (was only left
      in changelog)
    - ntpdate Recommends (was missing in former Changelog, now superseded
      by timesyncd which is available by default)
    - adding of d/docs for dh_installdocs

dovecot (1:2.2.24-1) unstable; urgency=medium

  * [26020b6] Imported Upstream version 2.2.24 (Closes: #818652)

dovecot (1:2.2.23-1) unstable; urgency=medium

  [ Jaldhar H. Vyas ]
  * Drop missing-expunges.patch, merged upstream

  [ Apollon Oikonomopoulos ]
  * [8a01915] Imported Upstream version 2.2.23

 -- Christian Ehrhardt <email address hidden> Tue, 21 Jun 2016 16:31:29 +0200

The dovecot-antispam rebuild was pushed along the new version - marking bug task accordingly.
FYI - https://launchpad.net/ubuntu/+source/dovecot-antispam/2.0+20150222-1build2

We discussed how to handle the change in Xenial.
We can't just "drop" the package - instead we have to make it an empty package.

Left in there should be a README and an entry in NEWS.Debian and obviously changelog to explain what&why happened.

I hope I find the time to preparing that today.

The package now looks like this:
drwxr-xr-x root/root 0 2016-09-21 11:12 ./
drwxr-xr-x root/root 0 2016-09-21 11:12 ./usr/
drwxr-xr-x root/root 0 2016-09-21 11:12 ./usr/share/
drwxr-xr-x root/root 0 2016-09-21 11:12 ./usr/share/doc/
drwxr-xr-x root/root 0 2016-09-21 11:12 ./usr/share/doc/dovecot-lucene/
-rw-r--r-- root/root 428 2016-09-21 10:43 ./usr/share/doc/dovecot-lucene/NEWS.Debian.gz
-rw-r--r-- root/root 86 2016-09-21 11:06 ./usr/share/doc/dovecot-lucene/README.Debian
lrwxrwxrwx root/root 0 2016-09-21 11:12 ./usr/share/doc/dovecot-lucene/changelog.Debian.gz -> ../dovecot-core/changelog.Debian.gz
-rw-r--r-- root/root 7487 2016-03-19 00:29 ./usr/share/doc/dovecot-lucene/copyright

Prepring SRU template, attach debdiff and upload into xenial queue for evaluation by the SRU Team

Finalized SRU template and ran the tests again on a container using the new upload (built locally).
Works as expected and is reflected in the [Test Case] section.

Uploaded, so setting Fix committed for now.

Hello Chris, or anyone else affected,

Accepted dovecot into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dovecot/1:2.2.22-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I doubt that anybody else jumps in to verify, so doing that on my own - in a new container it was working from proposed just as it should now.

This bug was fixed in the package dovecot - 1:2.2.22-1ubuntu2.1

---------------
dovecot (1:2.2.22-1ubuntu2.1) xenial; urgency=medium

  * Disable dovecot-lucene plugin as it had various issues, universe
    dependencies and is deprecated in favor of solr (LP: #1524526).
    - make dovecot-lucene an empty package.
    - disable building lucene, now --without-lucene.
    - remove libclucene-dev build dependency.
    - drop debian/dovecot-lucene.links, debian/dovecot-lucene.dirs and
      drop debian/dovecot-lucene.lintian-overrides.
    - add dovecot-lucene.README.Debian and dovecot-lucene.NEWS to explain
      to users why it is empty.

 -- Christian Ehrhardt <email address hidden> Wed, 21 Sep 2016 10:26:44 +0200

The verification of the Stable Release Update for dovecot has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

FYI - no more an issue in Cosmic (test build) so we will stop disabling dovecot-lucene there.