Comment 8 for bug 198368

Okay, after running a hacked maildir.py and watching what happens, I have a new hypothesis. The enhanced log shows that we are running out of file descriptors.

Maildir.newMessage() opens files and passes them to MaildirMessageWriter, but the latter does not close the files until commit() or abort(). I added some extra debugging and it's clear that for some list notifications, we're seeing tons of opens before we see a close. Well, what if we're trying to notify more people than we have file descriptors available, and every notification generates a new maildir file? Zope will just happily exhaust fds before it hits two phase commit or abort. This seems like a fundamental flaw somewhere <wink>.

I can think of a couple of ways out of this, though I'm not sure yet what's feasible in Zope. You could chunk up the transactions to a small number of recipients, and deliver to that chunk then commit(). This could mean that other errors will cause some recips to get notifications and others not, so we'd have to worry about any duplications during recovery.

Maybe we have to change the protocol so that the fd is closed sooner than commit() or abort(), since that's always going to happen anyway. The client of MaildirMessageWriter has to know when that will happen though.

Seems like we're probably going to have to patch Zope though to fix this.