Merge dovecot 2.3.x for Cosmic

Bug #1771524 reported by Christian Ehrhardt 
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
dovecot (Debian)
Fix Released
Unknown
dovecot (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

There are a few things that make this merge more uncommon than usual
1. Upstream has a new stable series 2.3 and are on 2.3.1 already.
   Debian is working on is, but still in experimental, but we'd like to do that for Cosmic.
   Therefore syncing on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=896674
2. a bunch of new build deps in Debian, but all are in main already
   libapparmor-dev, liblua5.3-dev, libsodium-dev, libicu-dev
3. We will drop mail-stack-delivery as what the addition provided is these days done
   a) already in base package (like ssl setup)
   b) otherwise in external projects e.g. mail-in-a-box
   c) Users might volunteer to maintainer what mail-stack-delivery was, but do so outside of the package

Related branches

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

ML Discussions:
https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/040264.html
https://lists.ubuntu.com/archives/ubuntu-devel/2018-May/040324.html

I already answered, but I can't find it in the archive - so I quote my mail here to be sure:
Hi Phillip, Serge and Mark,
I appreciate community interest in this - but from what I heard so far you'd be just as good with an out of tree solution not part of the actual (dovecot) packaging right?
The current mail-stack-delivery does modify config files for you, so you could drive the same via e.g. a github project just as Mark does with his mail server he mentioned.
If it finds a lot of followers it might become a next mail-in-a-box - and if not it is fine as well - a community is meant to live and try out things.
If you want you can start with what today's mail-stack-delivery does, but I think this is not required (maybe better from scratch).

@Phillip/Serge:
I don't know if you'd prefer a github project or similar that just provides a bit of scripting or if you explicitly want to maintain it as a package?
I'd think it is not needed, but I'm fine with the latter as well, but in that case I'd still want to take it out of the dovecot package this cycle and you can then push it as a universe community package (or as a snap) separate to that.
This would simplify the doveot packaging, but not prevent you from maintaining the same the way you want - so would that be ok for you as well?

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Also in a discussion with fellow Ubuntu server team members all were still +1 on dropping it and recommending to maintain such code outside of the package (if anyone wants)

Changed in dovecot (Debian):
status: Unknown → New
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Since 2.3.1 is still in experimental (see Debian bug) I moved the lp1771524/new/debian tag to that manually.

passed all Dovecot bugs of the last 2 years to clear out or consider on the merge.

Reconstruct, Deconstruct and Logical prepared in git merge workflow.

Checked CVE patches to exist in 2.3.1
- CVE-2017-15132 is already included
- CVE-2017-15130 is already included
- CVE-2017-14461 is already included

There were some conf-breaking changes in 2.3, so I consider it even more important to make it available early on Cosmic cycle as well as in the long run towards 20.04.

Note for mail-stack delivery:
- only a few breaks replaces will remain according to case #11 on https://wiki.debian.org/PackageTransition until after 20.04
- we will not rm-conffile the modifications it had made since it did modify "its own" but dovecots config, therefore removing would be an unwanted modification of a users configuration

All that said ready to rebase and drop a lot this time ...

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Dropped (by dropping mail stack delivery):
 276a97d3 2.2.22-1ubuntu Add mail-stack-delivery package
 d95bbb9a * Use ssl key paths set up by Debian as default - adapt mail-stack-delivery default config - move old mail-stack-deliver
 086f95d2 - for the ssl transition to work we need to ensure dovecot-core is complete before upgrading mail-stack-delivery
 fcfab6ea d/mail-stack-delivery.postinst: add SSL_CERT/SSL_KEY detection to postconf
 746601a4 fix SSL_CERT/SSL_KEY detection
 8f8a7d17 - debian/99-mail-stack-delivery.conf: drop explicit ssl_cipher_list, recent upstream has sane defaults now
 1e33020a - d/mail-stack-delivery.postinst: stop modifying mandatory tls config, recent upstream has sane defaults now
 07c4a590 d/mail-stack-delivery.preinst: improve moving of old mail-stack-delivery conf (merge with '* Use ssl key paths set up by Debian
 49d2e29e d/mail-stack-delivery.postinst: fix minor shell programming issues
 3078196f d/mail-stack-delivery.postinst: postfix conf to follow key changes
 bf4f82fc fix dput lint warnings in regard to dependencies

Droped (now upstream)
 d9dcfdf4 debian/patches/CVE-2017-15132.patch
 b971a03e debian/patches/CVE-2017-15132-additional.patch
 b468f56c debian/patches/CVE-2017-14461/*.patch: upstream parsing fixes
 572b4ec8 debian/patches/CVE-2017-15130/*.patch: upstream config filtering fix

Dropped (transition no more needed >18.04)
  4 pick 614a88a3 handle conffile removal of /etc/init/dovecot.conf

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Also libstemmer-dev is now in main
$ apt-cache policy libstemmer-dev
libstemmer-dev:
  Installed: (none)
  Candidate: 0+svn585-1build1
  Version table:
     0+svn585-1build1 500
        500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

So we can depend on that as well now.

Lets try to re-submit the tests to Debian?
TODO - ensure they work with the latest dovecot first

And also Lucene - maybe the old issue is fixed now.
TODO - recheck #1524526

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Checked bug 1524526 in a test build.
It is no more an issue there (reason unclear, but could be deep in build and ld).
Anyway, that means we really can stop dropping lucene.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I had thought of an issue of the breaks/replaces and tested with the first ppa build.
Indeed old mail-stack-delivery postrm being triggered would mean a restoration of the cofig before it's installation. This is not how we want it to be transitioned-out.
Therefore case 11 of https://wiki.debian.org/PackageTransition does not apply (as that would remove it).
Instead approach #2 would be to add an empty transitional that we will keep until 20.04 to catch all upgrades dropping it without modifying the configuration.

I tested an old install with mail-stack-delivery upgrading to the new version.
a) with breaks/replaces
   triggers the unwanted remove as expected
b) with a transitional package
   This does what we want
    - turn it into an empty package
    - does not rebuild the assumed old config
    - the transitional can be removed without wreaking havok

Going on with B then ...

tags: added: upgrade-software-version
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Changed in dovecot (Ubuntu):
status: New → In Progress
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Tested autopkgtests manually as doing so on the images is still broken.
The old test was centered around mail-stack-delivery and made no sense for Debian.
Now that we dropped that but e.g. default ssl is part of dovecot-core we might refresh the tests and then make them usable for Debian as well.

Most works without mail-stack-delivery just fine, so lets consider cleaning those tests up and proposing them to Debian.

But this is again a case showing us the limits and pain of redundancy.
These tests are maintained in git+ssh://git.launchpad.net/qa-regression-testing and only there.
Due to that the in package tests bit rot already, unless we want to move the tests fully (and only) into the package we won't succeed and always to work twice.

I'll fix the issues in the upstream (test) repo, but drop the tests from the package.
The package already has tests, and those we carry here we run in addtion to the autopkgtests on major changes (like an SRU or a Merge).

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Auth-Fails in almost all cases of imap and pop3 login.
Already this fails:
$ telnet localhost 110
user ubuntu
pass ubuntu

Dovecot should use the passwd, according to the config the test sets up.

Gets:
auth[2573]: PAM audit_log_acct_message() failed: Operation not permitted

No associated apparmor deny or such, no other journal entry.
I debugged and found plenty of data, finally after I knew what I was up to I found:
- https://bugzilla.redhat.com/show_bug.cgi?id=1564348
- https://github.com/dovecot/core/pull/71

This will likely become rather different later on, but for now lets mask the critical new entries in the service file.

I added a commit for that to the MP and rebuild the PPA.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Due to many instabilities in 2.3.1 (found on the mailing list) we might do a merge of a more stable version for now.

I'm postponing the MP of 2.3.1 until I have synced with Debian and see a few more issues resolved.

Changed in dovecot (Ubuntu):
status: In Progress → Incomplete
summary: - Merge newer dovecot for Cosmic
+ Merge dovecot 2.3.x for Cosmic
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Subscribed ubuntu-server to take a look at this again later this or next cycle.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

2.3.2 was released (even 2.3.2.1) and in Debian.
Time to reconsider 2.3.x

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Ok, looks good in tests now - pushing changes
That was a loooon loop just to eventually drop them :-/

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

That comment was meant for the MP, sorry for the noise :-)

Changed in dovecot (Ubuntu):
status: Incomplete → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dovecot - 1:2.3.2.1-1ubuntu1

---------------
dovecot (1:2.3.2.1-1ubuntu1) cosmic; urgency=medium

  * Merge with Debian unstable (LP: #1771524). Remaining changes:
    - carry mail-stack-delivery as empty transitional package
      (can be dropped >20.04)
  * Dropped Changes
    - Add updated autopkgtest to debian/tests/* (LP: 1638865)
      (those becamce superfluous by being mostly identical to the tests in
       debian/tests/usage that are now packaged by Debian.)

 -- Christian Ehrhardt <email address hidden> Fri, 03 Aug 2018 12:31:43 +0200

Changed in dovecot (Ubuntu):
status: In Progress → Fix Released
Changed in dovecot (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.