lmtp crashes with: Panic: file mail-user.c: line 229 (mail_user_deinit): assertion failed: ((*user)->refcount == 1)

Status changed to 'Confirmed' because the bug affects multiple users.

Hi Andre,

Thank you for your report and that you help us with reports.
Could you please provide steps to reproduce that bug?
I will take care of that from Monday.

Thank you

Unfortunately I don't know how to reproduce this problem. It seems to happen with some specific customer accounts, but I'm unable to reproduce the error by configuring them in Thunderbird. It's possible that it only happens for some specific IMAP client.

What I can say is that I've backported the package from Kinetic, which has Dovecot 2.3.19, and the error is fixed for me.

Hi Andre,
If we can not identify steps to recreate and verify this case, can we count on you and your users to verify the packages as they would be serviced?
If so, we could provide a PPA with the package and once confirmed to work fine, we could prepare an SRU upload to Jammy to then have your verification again.
Do you think that you can help us in that case and provide that?

Yes, absolutely!

Hi Andre,

I've prepared a PPA package for you. Please remember that it's only for testing purposes and not intended for anything else.
https://launchpad.net/~michal-maloszewski99/+archive/ubuntu/jammy-dovecot-ppa-1992848
Here is a link which leads to the PPA and there you have a instruction how to add this PPA to your system.
Thank you in advance for your help.

Hi Michał

Unfortunately the error remains when using the packages from your PPA. It seems that the real fix must have been introduced by some other commit in the 2.3.16-2.3.19 period.

Hi Andre,

Thanks for your feedback. I have checked the PPA and my dsc package separately whether I missed something. Nothing is missing.
I have a question for you.
Could you please send me some logs of what happens when you bump into the issue? It'd be helpful.
In addition, please check if you did perform 'apt update' after you've added PPA to your system.
It would also help if you showed me the apt-get logs.
These are the next steps. I will try to find another fix which might be helpful, but maybe you have an extra idea what else might help here as you did at the beginning in the bug report (another commit/fix).

The log message is the same as the one I posted originally. There is also a backtrace, which I didn't include in the original message.

Oct 21 14:04:05 imap1 dovecot: lmtp(1797565): Panic: file mail-user.c: line 229 (mail_user_deinit): assertion failed: ((*user)->refcount == 1)
Oct 21 14:04:05 imap1 dovecot: lmtp(1797565): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x46) [0x7f655cc53d76] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7f655cc53e92] -> /usr/lib/dovecot/libdovecot.so.0(+0x109c6b) [0x7f655cc60c6b] -> /usr/lib/dovecot/libdovecot.so.0(+0x109d07) [0x7f655cc60d07] -> /usr/lib/dovecot/libdovecot.so.0(+0x5cf53) [0x7f655cbb3f53] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x483c5) [0x7f655cd753c5] -> dovecot/lmtp [10.20.32.185 DATA](lmtp_local_data+0x3e6) [0x564ff3232996] -> dovecot/lmtp [10.20.32.185 DATA](client_default_cmd_data+0x191) [0x564ff32314c1] -> dovecot/lmtp [10.20.32.185 DATA](cmd_data_continue+0x180) [0x564ff3231250] -> /usr/lib/dovecot/libdovecot.so.0(+0x7c656) [0x7f655cbd3656] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x6d) [0x7f655cc76f1d] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x13a) [0x7f655cc7861a] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x54) [0x7f655cc76fc4] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x7f655cc77180] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x17) [0x7f655cbe96d7] -> dovecot/lmtp [10.20.32.185 DATA](main+0x225) [0x564ff322fec5] -> /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f655c951d90] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f655c951e40] -> dovecot/lmtp [10.20.32.185 DATA](_start+0x25) [0x564ff322ffc5]

These are the dpkg logs. I have installed the packages manually because due to network policy reasons the Dovecot server cannot access external repositories.

2022-10-21 12:47:32 upgrade dovecot-core:amd64 1:2.3.19.1+dfsg1-2ubuntu2 1:2.3.16+dfsg1-3ubuntu2.2~ppa1
2022-10-21 12:47:32 status half-configured dovecot-core:amd64 1:2.3.19.1+dfsg1-2ubuntu2
2022-10-21 12:47:32 status unpacked dovecot-core:amd64 1:2.3.19.1+dfsg1-2ubuntu2
2022-10-21 12:47:32 status half-installed dovecot-core:amd64 1:2.3.19.1+dfsg1-2ubuntu2
2022-10-21 12:47:32 status triggers-pending man-db:amd64 2.10.2-1
2022-10-21 12:47:32 status unpacked dovecot-core:amd64 1:2.3.16+dfsg1-3ubuntu2.2~ppa1
2022-10-21 12:47:32 upgrade dovecot-imapd:amd64 1:2.3.19.1+dfsg1-2ubuntu2 1:2.3.16+dfsg1-3ubuntu2.2~ppa1
2022-10-21 12:47:32 status half-configured dovecot-imapd:amd64 1:2.3.19.1+dfsg1-2ubuntu2
2022-10-21 12:47:32 status unpacked dovecot-imapd:amd64 1:2.3.19.1+dfsg1-2ubuntu2
2022-10-21 12:47:32 status half-installed dovecot-imapd:amd64 1:2.3.19.1+dfsg1-2ubuntu2
2022-10-21 12:47:32 status triggers-pending ufw:all 0.36.1-4build1
2022-10-21 12:47:32 status unpacked dovecot-imapd:amd64 1:2.3.16+dfsg1-3ubuntu2.2~ppa1
2022-10-21 12:47:32 upgrade dovecot-lmtpd:amd64 1:2.3.19.1+dfsg1-2ubuntu2 1:2.3.16+dfsg1-3ubuntu2.2~ppa1
2022-10-21 12:47:32 status half-configured dovecot-lmtpd:amd64 1:2.3.19.1+dfsg1-2ubuntu2
2022-10-21 12:47:32 status unpacked dovecot-lmtpd:amd64 1:2.3.19.1+dfsg1-2ubuntu2
2022-10-21 12:47:...

I've marked this Fix Released as Kinetic is on 2.3.19 so this is presumably fixed there, and have opened a task for Jammy to track that the bug needs fixing on Jammy.

Thank you for the details. I think there must be another upstream change that we need to cherry-pick then? We'd be happy to pull it into Jammy for you, but we need some help please, starting with identifying the specific fix. Unfortunately we need to rely on you for this please since we don't know how to reproduce the issue.

Hello,

I'm also affected by this issue as of about 1/12/23

It repeats for every attempt of delivery via Dovecot. I did an 'apt upgrade' and restart with no resolution. My most recent logs are below. I can't receive email so I'll check back on this issue through the website.

I'm running Jammy on a VPS with Postfix as my MTA and Dovecot as my MDA/LDA. Email has worked for the last 6 months on this VPS and for years using a similar stack. This issue came up only within the last few days. The messages are being marked as deferred to the senders but Postfix is receiving them and deferring because Dovecot dies partway through the delivery process.

Another note, this seems to only affect my primary email address. Other accounts are able to receive email from internal and external servers without issue.

Jan 14 10:15:34 dovecot: lda(x@x.com)<4054740><9bvcGZb6o2XU3j0Ar6IjXQ>: Panic: file mail-user.c: line 229 (mail_user_deinit): assertion failed: ((*user)->refcount == 1)

Jan 14 12:19:15 dovecot: lda(x@x.com)<2599><H/5+CpMXpGUnCgAAr6IjXQ>: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x46) [0x7fb2a5916de6] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7fb2a5916f02] -> /usr/lib/dovecot/libdovecot.so.0(+0x109cdb) [0x7fb2a5923cdb] -> /usr/lib/dovecot/libdovecot.so.0(+0x109d47) [0x7fb2a5923d47] -> /usr/lib/dovecot/libdovecot.so.0(+0x5cf73) [0x7fb2a5876f73] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x483c5) [0x7fb2a5a383c5] -> /usr/lib/dovecot/deliver(main+0x938) [0x55b189b3c628] -> /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb2a5613d90] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fb2a5613e40] -> /usr/lib/dovecot/deliver(_start+0x25) [0x55b189b3cc95]

Hi Brandon,

Thanks for the report.

How often does the issue occur?

Can you reliably reproduce the issue? If so, would you mind providing us a short reproducer?

Finally, did you have the chance to try the proposed fix in the PPA provided in https://launchpad.net/~michal-maloszewski99/+archive/ubuntu/jammy-dovecot-ppa-1992848?

The issue was happening for every email I received for a certain account.

I installed the proposed fix from the PPA and it seems to be resolved now.

Thank you for the information! If there's anything else I can provide I am available to assist. Like if you want me to remove the PPA and do some add'l testing.

With the PPA installed will that leave my dovecot install in an un-updatable state though?

This is interesting.

Brandon, would you be willing to move back to the package version in the jammy archive to ensure the problem is back when you do so (1:2.3.16+dfsg1-3ubuntu2.2)

Also, do you think you can reproduce the issue with a similar configuration as the one you have set for that account?

My current concern here is that Andre reported that the ppa did not fix the issue for them. But now it does seem to fix it for you.

Without a reproducer, we would need to rely on you and Andre to ensure the issue is fixed.

Andre, I see from your dpkg logs that your dovecot packages were left in a half-configured state. would you be able to provide additional logs?

In the mean time, I wonder if we should proceed with this SRU in case Brandon confirms they can reliably trigger the bug when they move back to the jammy package version and it is indeed fixed when using the provided PPA.

Hello Athos,

I don't mind reproducing the issue.

Can you give me instructions on how to remove and re-add the ppa?
That and a list of my config files that you'll need should be enough for me to:

1) Revert the ppa to make Dovecot broken again
2) Attempt to send an email to the broken account
3) Capture logs from /var/log/mail.log
4) Install the ppa again
5) Confirm that emails to the previously broken account are received
6) Send Dovecot configs to this thread

I think that should be everything but please confirm

--Brandon

Hi Brandon! The newest version of dovecot is actually 1:2.3.16+dfsg1-3ubuntu2.2 in the main archives, which is a higher version number than the PPA now, so if you just upgrade like regular you should see the package upgrade. There's no need to really remove the PPA from your system, as we can install the right version with apt. Right now you should be able to see this

```
$ sudo apt update
$ sudo apt-cache policy dovecot-core
dovecot-core:
  Installed: 1:2.3.16+dfsg1-3ubuntu2.2~ppa1
  Candidate: 1:2.3.16+dfsg1-3ubuntu2.2
  Version table:
     1:2.3.16+dfsg1-3ubuntu2.2 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
 *** 1:2.3.16+dfsg1-3ubuntu2.2~ppa1 500
        500 https://ppa.launchpadcontent.net/michal-maloszewski99/jammy-dovecot-ppa-1992848/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     1:2.3.16+dfsg1-3ubuntu2.1 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     1:2.3.16+dfsg1-3ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
```

The 3 *'s show which version is currently installed.

So at this point you should be able to just run the following command to update to the latest dovecot

$ sudo apt install dovecot-core

Now to get to the older 2.1 version we can also do that by the following command

$ sudo apt install dovecot-core=1:2.3.16+dfsg1-3ubuntu2.1 --allow-downgrades

It would be handy (if you are willing) to see how both versions of dovecot work for you.

Hi Andre and Brandon,

It looks like there was some waiting for feedback from you, but I've proceeded with preparing the next steps towards a Stable Release Update for the fix, since it looks pretty straightforward. Optionally, you may want to review past comments and respond to earlier questions, in case the additional feedback helps in deciding whether to proceed with this as an SRU.

I've created a new PPA you can use for testing, here:
   https://launchpad.net/~bryce/+archive/ubuntu/dovecot-sru-lp1992848

It should be essentially the same as Michal's just with an updated changelog entry.

Meanwhile, the next required action will come after the SRU team has looked at this and determined if they'll accept it into the jammy-proposed pocket for verification. You should expect to see a reply on this bug with their decision and/or directions.

Note that since a reproducing test case was not identified and defined, my team will be unable to perform the validation step -- we'll be relying on you to verify that you still reproduce the issue without this update, and that it goes away WITH this update, and furthermore that no additional regressions show up.

I don't envision there will be any problems; the change itself is quite modest and already applied upstream. But please keep a close eye just in case something weird happens, and please follow up to the SRU team's testing requests to this bug when they are posted.

Hello Andre, or anyone else affected,

Accepted dovecot into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dovecot/1:2.3.16+dfsg1-3ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.