Launchpad discards email non delivery reports

Really, we should hook into or steal Mailman's logic here, as it does all the right-ish things with VERP and bounce detection.

We are hoping that we can steal the logic more easily once Barry migrates us to Mailman 3, which he has penciled in as 1Q 2011.

Marking critical at Robert's request.

We're under discussion for refactoring certain parts of Mailman 3 for use here.

Barry and I talked about how to tackle this. The following are notes on what we hashed out. All the good stuff is from Barry.

Barry is factoring out a library from Mailman 3 to help with this now. It will determine whether a given email.message.Message is a bounce message, and if so, it will return the email addresses involved and whether the bounce is temporary or permanent.

It is expected to work within a scenario like the following. Launchpad sends VERP-formatted emails. We have a custom LMTP server that gets bounces and (using Barry's library) figures out what's going on. It sends that information to Launchpad, which records the bounce information. A cronscript processes the Launchpad bounce information, and among other things, calculates how the bounces should affect the given user email addresses. A change to the email page allows people to re-enable disabled email addresses, possibly among other changes.

Coordination with IS:

* EARLY
 - verify VERP format
 - agree on feasibility of running a custom LMTP server in production, and of running more if we need to scale; these need to be able to contact a given port
 - as above for staging and qastaging
 - discover current rate of bounce emails sent to Launchpad
* LATE
 - Install custom LMTP server
 - Install EXIM rules to send all Launchpad bounce messages (including those with VERP addresses) to the LMTP server(s)

Details on other components and changes:

* Create an LMTP server (the stdlib has one in smtpd). It accepts messages and calls Barry's library (see above) with them. For each message that the library identifies as a bounce, it contacts Launchpad via some protocol and port and format (e.g., xmlrpc) to deliver information about it. This information may include the email addresses that bounced; whether the bounce is temporary or permanent; and the bounce message.
* When Launchpad receives a call from the LMTP service, it records the message id, the date, the bounced email addresses, whether the bounce is permanent or temporary, and possibly a context (e.g., the source of the mail that caused the bounce, such as a bug notification), and possibly the message body (to be able to answer user queries; Barry said that Mailman did not do this but people wanted it).
* A cron (or similar) runs to do the following tasks:
 - process bounce events, converting them into bounce scores for each address. A bounce score increases by one for every day that gets one or more permanent bounces.
 - Decide if an email address should be disabled (e.g., score > 5). If disabled email address is preferred, choose another email to be preferred, if there is one.
 - Send probes to disabled emails once a week for N (4?) weeks informing them that the address is disabled, and sending them to the web email page to re-enable.
 - Set the bounce score to zero for email addresses that have not increased their score for N (7?) days and that are not disabled.
* Change the email web page for users and admins to deactivate and reactivate emails
* Change sending emails from Launchpad in two ways.
 - First, they should include VERP information (e.g., "from = 'foo-bounces+gary.poster=canonical.com@launchpad....

This sounds great. I have a few suggestions.

Internal protocol: using the internal private-xmlrpc service for
telling LP whats going on is a great idea.

Tables etc - whatever works, but consider bringing this up as a
dedicated microservice. Perhaps thats not feasible, but this seems
clearly stubby, able to be upgraded or migrated without user visible
downtime, maintaining state mostly irrelevant to the rest of the
system : a *great* situation for a microservice.

-Rob

I have a first cut at a refactored bounce detection library. See lp:flufl.bounce

It's a pretty standard setuptools library, and even passes its tests (for me :) using `python setup.py test`. I'm marking it as 0.9 for now and not quite releasing it, though I'd like to soon.

If you would, please take a look at the branch and let me know if the basic API will do what you want. I'm still considering building up some of the higher level stuff in the above outline, but for now, the detectors work pretty well and are ready for refactoring Mailman 3 to use them. Take a look at flufl/bounce/docs/using.rst for details.

Thanks, Barry! I hope that we'll start playing with this within a week or two.

On a more general note, see also bug 592814, for an ISD use case for something similar. As Stuart M said there, it would be great to be able to have something that suits both our needs, and this general idea sounds like it might be a good direction for that.

I have a (probably fairly obvious) sketch in my head of what could be pushed out to a microservice from what I wrote in comment #5. Basically, everything except for the concept of a disabled email, and the management of disabled emails, could probably be pushed out to a microservice. I'll write it out later if no one beats me to it.

Sadly, I don't have access to bug 592814. Can you subscribe me to that, and will that give me access? ;) If not, please add a comment about flufl.bounce; I'm of course happy to make sure the library serves a wide range of purposes.

Sorry for not being clear, Barry. I meant that the full aggregation described in comment 5 (the LMTP server and even the cronjob-y bits that would be part of the bounce "service") might be of value to them, not just flufl.bounce.

On Jul 06, 2011, at 08:42 PM, Gary Poster wrote:

>Sorry for not being clear, Barry. I meant that the full aggregation
>described in comment 5 (the LMTP server and even the cronjob-y bits that
>would be part of the bounce "service") might be of value to them, not
>just flufl.bounce.

Indeed. I think done properly, it could be a generally useful application.
Let's work out the licensing/copyright issues in advance. I think
GPLv3/LGPLv3 would be more appropriate than AGPL.

r9 of flufl.bounce passes tests for me. Thanks, Barry! I'll look more into it today.

Barry said:
> Let's work out the licensing/copyright issues in advance. I think
> GPLv3/LGPLv3 would be more appropriate than AGPL.

After discussing this, I think Barry and I agreed that the application would start under our usual AGPL license; if we determined that code could be factored out as a library, we would make a request at that time to use LGPL for those parts.

RT 46793 tracks pertinent pre-imp questions for LOSAs.

https://dev.launchpad.net/LEP/BounceHandling

Evaluated with Francis. This is too big for a squad on bug rotation. This will need to be scheduled. The LEP has the work so far.