mutt: opens of same mbox by multiple mutt processes erase message bodies

Automatically imported from Debian bug report #330474 http://bugs.debian.org/330474

Message-Id: <email address hidden>
Date: Wed, 28 Sep 2005 04:13:48 -0500
From: Branden Robinson <email address hidden>
To: Debian Bug Tracking System <email address hidden>
Subject: mutt: opens of same mbox by multiple mutt processes erase message bodies

Package: mutt
Version: 1.5.11-1
Severity: critical

This is awful.

For years I have performed multiple opens of mbox files, using multiple
mutt processes.

For years it worked.

Now it doesn't.

Now, if you do this, message bodies will be nuked out of the folder and the
Content-Length: header rewritten to zero.

The latter appears to be a subtly different issue; the header was rewritten
for some but not all messages, and the ones with the nuked bodies were a
*subset* of the ones whose Content-Length: headers were written.

This is very, very, very, very bad behavior, and I would be deeply furious
about it if I hadn't been able to use e2undel on the remote host where my
mail spools to recover the ~150MB deleted spool file from a live filesystem
and extract the obliterated message bodies from it. I was miraculously
lucky. Anybody who spools mail locally, doesn't have root on the host
where their mail spools, or who uses ext3 would have been badly screwed.

Mutt should either go back to not destroying mailbox contents on multiple
opens, should implement a locking mechanism to prevent other copies of
itself from opening a given folder for writing at the same time, or
(easiest to implement but least useful), refuse to open a mailbox that
another Mutt process has open.

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'stable')
Architecture: powerpc (ppc)
Shell: /bin/sh linked to /bin/bash
Kernel: Linux 2.6.9-powerpc-smp
Locale: LANG=C, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)

Versions of packages mutt depends on:
ii libc6 2.3.5-6 GNU C Library: Shared libraries an
ii libdb4.3 4.3.28-3 Berkeley v4.3 Database Libraries [
ii libgnutls12 1.2.6-1 the GNU TLS library - runtime libr
ii libidn11 0.5.18-1 GNU libidn library, implementation
ii libncursesw5 5.4-9 Shared libraries for terminal hand
ii libsasl2 2.1.19-1.6 Authentication abstraction library
ii postfix [mail-transport-agent 2.2.4-1 A high-performance mail transport

Versions of packages mutt recommends:
ii locales 2.3.5-6 GNU C Library: National Language (
ii mime-support 3.35-1 MIME files 'mime.types' & 'mailcap

-- no debconf information

Message-ID: <email address hidden>
Date: Wed, 28 Sep 2005 14:48:46 +0200
From: Adeodato =?utf-8?B?U2ltw7M=?= <email address hidden>
To: Branden Robinson <email address hidden>, <email address hidden>
Subject: Re: Bug#330474: mutt: opens of same mbox by multiple mutt processes erase message bodies

* Branden Robinson [Wed, 28 Sep 2005 04:13:48 -0500]:

> For years I have performed multiple opens of mbox files, using multiple
> mutt processes.

> For years it worked.

> Now it doesn't.

  As mentioned on IRC:

    (a) I can't reproduce this after a quick test, including the
        procedure you mentioned there (open, reply to a message, open in
        another terminal and view that message).

        A description of how to reproduce would be good.

    (b) If you can rebuild mutt, please try after removing cvs_2005-09-24
        in the upstream/patches/ directory.

  Cheers,

--
Adeodato Simó
    EM: asp16 [ykwim] alu.ua.es | PK: DA6AE621

Hanlon's Razor: Never attribute to malice that which is adequately
explained by stupidity.

Message-ID: <email address hidden>
Date: Thu, 29 Sep 2005 11:03:06 -0500
From: Branden Robinson <email address hidden>
To: <email address hidden>, <email address hidden>
Subject: Re: Bug#330474: mutt: opens of same mbox by multiple mutt processes erase message bodies

--ikeVEW9yuYc//A+q
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

retitle 330474 mutt: rewrites Content-Length message header in mbox folders=
 to be zero, which subsequently causes mutt to nuke the entire message body
thanks

My initial diagnosis of this bug was overly complex.

Here's a repro recipe.

1) Open any mbox folder with "mutt -R". (-R means "read-only", you don't
have to do this, but it will allow you to observe the bug without
destroying your mail folder).

2) Tag one or messages in that folder.

3) Observe the Content-Length headers in the message(s) you've tagged. In
most cases of real-world mail, the header value will be non-zero.

4) C)opy the tagged message(s) to a new folder.

5) Quit.

6) Use $PAGER or $EDITOR to view the new folder.

7) Observe the Content-Length headers in the message(s). Note that they
are all zero.

8) Observe that the message body (or bodies) is/are still present.

9) Open the new folder with mutt. Don't specify -R this time.

10) Read the message(s) in it.

11) Observe that you can't read the message body.

12) Quit back to the index, and quit mutt. Don't use "exit". We want
changes written to the folder.

13) Use $PAGER or $EDITOR to view the new folder again.

14) WHERE THE HELL ARE THE MESSAGE BODIES?

Retitled bug accordingly.

I stand by the severity of this report.

I don't know if it's specific to mbox folders; that's just the only format
I ever use.

--=20
G. Branden Robinson | You are not angry with people when
Debian GNU/Linux | you laugh at them. Humor teaches
<email address hidden> | them tolerance.
http://people.debian.org/~branden/ | -- W. Somerset Maugham

--ikeVEW9yuYc//A+q
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iEYEARECAAYFAkM8EDkACgkQ6kxmHytGonw7PgCfcLzt4Vh7EwWLTfp+mVxKWRAd
xgAAoKuh+Dc9MIbVXvSUACkHIhWO7Gbg
=w5I6
-----END PGP SIGNATURE-----

--ikeVEW9yuYc//A+q--

Message-Id: <email address hidden>
Date: Thu, 29 Sep 2005 23:24:30 +0200
From: New Mutt PR <email address hidden>
To: Mutt Developers <email address hidden>,<email address hidden>,<email address hidden>
Subject: mutt/2096: eats mail by setting content-length to zero in mboxes

>Number: 2096
>Notify-List: <email address hidden>
>Category: mutt
>Synopsis: eats mail by setting content-length to zero in mboxes
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: mutt-dev
>State: open
>Keywords:
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Thu Sep 29 23:24:29 +0200 2005
>Originator: Branden Robinson
>Release: CVS 2005-09-24
>Organization:
>Environment:
>Description:
This is Debian Bug#330474, filed at severity critical, which you can read in http://bugs.debian.org/330474. In summary:

- reproducable on, at least, powerpc (but not on i386)
- loosing mail occurs in two steps:
  - a mailbox is opened, and then the Content-Length of its messages is set to 0
  - the mailbox is reopened; if there's modifications to it, the messages with Content-Lenght set to 0 get truncated

Se the How-To-Repeat section.
>How-To-Repeat:
Any mbox should do, but I've used this one-message-only one: http://people.debian.org/~adeodato/tmp/2005-09-29/mutt-eating-mail/mbox.gz

mutt -f mbox
<ENTER>
qq
cat mbox
# notice the Content-Length header
mutt -f mbox
wn
q
cat mbox
>Fix:
Reverting the introduction of fseeko/ftello fixes (http://people.debian.org/~adeodato/tmp/2005-09-29/mutt-eating-mail/revert-this.diff), but it would be better to investigate and come with a real fix.
>Add-To-Audit-Trail:

>Unformatted:

Message-Id: <email address hidden>
Date: Thu, 29 Sep 2005 23:27:50 +0200
From: Adeodato Simó <email address hidden>
To: <email address hidden>
Subject: bug 330474 is forwarded to http://bugs.mutt.org/2096

# Automatically generated email from bts, devscripts version 2.9.7
forwarded 330474 http://bugs.mutt.org/2096

Message-Id: <email address hidden>
Date: Fri, 30 Sep 2005 01:07:43 +0200
From: Brendan Cully <email address hidden>
To: Mutt Developers <email address hidden>,<email address hidden>,<email address hidden>
Subject: Re: mutt/2096: eats mail by setting content-length to zero in mboxes

Synopsis: eats mail by setting content-length to zero in mboxes

State-Changed-From-To: open->closed
State-Changed-By: brendan
State-Changed-When: Fri, 30 Sep 2005 01:07:43 +0200
State-Changed-Why:
Fixed in CVS, sorry about that.

**** Comment added by brendan on Fri, 30 Sep 2005 01:07:43 +0200 ****

Message-Id: <email address hidden>
Date: Fri, 30 Sep 2005 01:23:33 +0200
From: Adeodato Simó <email address hidden>
To: <email address hidden>
Subject: tagging 330474

# Automatically generated email from bts, devscripts version 2.9.7
tags 330474 fixed-upstream

Message-Id: <email address hidden>
Date: Thu, 29 Sep 2005 17:02:13 -0700
From: =?utf-8?q?Adeodato_Sim=C3=B3?= <email address hidden>
To: <email address hidden>
Subject: Bug#330474: fixed in mutt 1.5.11-2

Source: mutt
Source-Version: 1.5.11-2

We believe that the bug you reported is fixed in the latest version of
mutt, which is due to be installed in the Debian FTP archive:

mutt_1.5.11-2.diff.gz
  to pool/main/m/mutt/mutt_1.5.11-2.diff.gz
mutt_1.5.11-2.dsc
  to pool/main/m/mutt/mutt_1.5.11-2.dsc
mutt_1.5.11-2_i386.deb
  to pool/main/m/mutt/mutt_1.5.11-2_i386.deb
mutt_1.5.11-2_powerpc.deb
  to pool/main/m/mutt/mutt_1.5.11-2_powerpc.deb

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to <email address hidden>,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Adeodato Simó <email address hidden> (supplier of updated mutt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing <email address hidden>)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.7
Date: Fri, 30 Sep 2005 01:15:28 +0200
Source: mutt
Binary: mutt
Architecture: i386 powerpc source
Version: 1.5.11-2
Distribution: unstable
Urgency: low (but fixes critical bug not in testing)
Maintainer: Adeodato Simó <email address hidden>
Changed-By: Adeodato Simó <email address hidden>
Description:
 mutt - Text-based mailreader supporting MIME, GPG, PGP and threading
Closes: 330474
Changes:
 mutt (1.5.11-2) unstable; urgency=low (but fixes critical bug not in testing)
 .
   * The fix for coping with mboxes bigger than 2 GB introduced a bug affecting
     at least powerpc (but not i386) which made mutt write Content-Length: 0 in
     mboxes due to a un-updated %ld format specifier. This caused for mail to
     be lost in the next mbox write.
 .
     Apply a patch quickly provided by upstream (thanks, Brendan Cully!) that
     makes mutt use the right format specifier. (Closes: #330474)
 .
   * Update the compressed folders to the 1.5.11, which includes documentation
     in XML format.
Files:
 16faa328d1f329f13fa319547c2fb240 330992 mail standard mutt_1.5.11-2.diff.gz
 50bbdb823bab04b6125bb48ce1e4b24a 1723536 mail standard mutt_1.5.11-2_powerpc.deb
 68b6361e2ed9bfc9398db7e8324e7586 787 mail standard mutt_1.5.11-2.dsc
 ceb76de244d36fcef0356e3fd0f20611 1682150 mail standard mutt_1.5.11-2_i386.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Signed by Adeodato Simó <email address hidden>

iEYEARECAAYFAkM8efIACgkQgyNlRdHEGII/6ACg4hj0cdIFZPEzVrbmNtZ2H7yc
XaoAn3ItS22SppQ6bvS3B1nGDMhVZbcF
=2O54
-----END PGP SIGNATURE-----

Fix released per:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=330474

Closing this bug/flea. Goodbye !