Loses e-mails when moving accross IMAP folders

Thank you for your bug. That's weird. Are you sure the mails are dropped? Maybe you don't display them all or something? Do you have easy steps to trigger the bug? Do you move them and clean the source while it's working or something? Evolution works fine for me and if it was deleting random mails we would have other bugs about it already

Yes, it is fully reproducible, even if random. You cannot move 20 mails without losing any, and sometimes even moving one at a time leads to loss.

Maybe the problem is related to Courier IMAP, but other clients on the same server, and even from the same client do not show this behaviour. In case you may wonder, the configuration on this IMAP server is pretty standard (for the version on Debian Etch).

Also, it is not specific to a particuliar computer, the problem shows on at least 2 or 3 different ones (2 laptops and one workstation, all x86).

How could I help you to pinpoint or even just reproduce this?

Maybe you could read http://www.gnome.org/projects/evolution/bugs.shtml and get a debug log?

We are closing this bug report as it lacks the information, described in the previous comments, we need to investigate the problem further. However, please reopen it if you can give us the missing information and don't hesitate to submit bug reports in the future.

I'm sorry, but I'm going tomorrow by the end-users to gather more information.
Please reopen it until Friday.

Here is more info:

I've changed the IMAP server to Dovecot, and the exact same problem occurs, no better, no worst.
I started Evolution with CAMEL_DEBUG=1, and when it occurs, I get the following error message:
(evolution-2.10:8724): camel-WARNING **: camel_exception_get_id calledwith NULL parameter.

(evolution-2.10:8724): camel-imap-provider-CRITICAL **:imap_command_start: assertion `store->ostream!=NULL' failed

Here, we just sent a mail, and we lost the copy which was supposed to be in the "Sent" folder (it was never filed).

Using Wireshark on the workstation, we could then see that no trafic was gone to the server (using IMAP protocol) when the problem occurs, while we could see connections when it does not fail to copy to Sent.

Please do not hesitate to ask for more information. If I need to go back to the users, I will.

status confirmed

--
Jérôme Warnier
FLOSS Consultant
http://beeznest.net

I reopen the bug as the reporter provided more technical information.

Jerome, could you please run with CAMEL_DEBUG=all instead of CAMEL_DEBUG=1? 'All' will give us more data. After the run please attach the output here -- at least a continuous segment from a bit before the problem to a bit after the problem.

I can reproduce this bug. In my case the mails are not lost, they end up in the local trash folder. By that I mean the second trash folder in an IMAP account folder list (the one that serves no useful purpose normally, has an icon). Not the one actually on the IMAP server, nor the one listed under "On this Computer".

Same setup: feisty, evo 2.10.1, imap server. Debug log attached.

I came across bug report this while searching for bug reports about a slightly different problem, namely sent emails not making it to the sent items folder on an IMAP server. There is an upstream bug report that suggests they may be related:

http://bugzilla.gnome.org/show_bug.cgi?id=436802

Whilst reviewing this log this morning I noticed something very strange at line 212:

received: * LIST (\Marked \HasNoChildren) "/" "Trash/Sent Items"

I don't have a "Sent Items" folder in any of my trash folders.

The message I moved to reproduce the bug is listed at line 618.

Some further info:

I sent myself several emails from another account for testing purposes and then proceeded to drag them back and forth between various folders until I triggered the bug again.

I observed that a move operation in Evo consists of a copy followed by a delete. Every time a mail is successfully copied, the original is moved to the trash folder. I had never noticed this behaviour with Evo before, I am sure in the past that the originals must have been immediately expunged. I don't recall my trash folder filling up with copies of messages I had moved. Is this in itself a separate bug?

On the occasion that the move operation failed Evo still moved the originals to the trash folder. Evo is failing to do the copy part of the operation but still continuing to move the originals to the trash folder! I guess the "bug" of not immediately expunging the originals should be viewed as a feature under such circumstances :-)

The mail account in question is with blueyonder.co.uk (now virginmedia). They use Microsoft SMTPSVC. I have another IMAP account with webmail.us and have never experienced this bug with them. They run Dovecot. Is this an IMAP server workaround that has recently broken perhaps? Might be interesting to know which mail server the original bug reporter is connecting to.

Couple of other things, just to clear up any confusion:

I found the missing "Trash/Sent Items" folder listed in the log file by configuring Evo to show all folders, not just subscribed folders. In actual fact this is a default folder created by blueyonder, I guess at some point in the past I must have mistakenly tried to delete it; it has now been undeleted and resubscribed. I can still reproduce this bug though, so I don't think this was in any way related to the problem.

In comment #10 I described the local trash folder as having no useful purpose. It is of course used by Evo, its just not the real trash folder on the imap server that the webmail interface or other mail clients like thunderbird use.

Simon Hepburn wrote:
> Some further info:

[..]

> The mail account in question is with blueyonder.co.uk (now virginmedia).
> They use Microsoft SMTPSVC. I have another IMAP account with webmail.us
> and have never experienced this bug with them. They run Dovecot. Is this
> an IMAP server workaround that has recently broken perhaps? Might be
> interesting to know which mail server the original bug reporter is
> connecting to.
It was Courier IMAP (4.1.1.20060828-5 from Debian Etch), and is now
Dovecot (now 1.0.rc15-2etch) (it seemed like a good idea while trying to
solve the problem, but it did not).
Please note this information was already in the bug report...

[..]

Sorry, I should have re-read the entire bug report before posting. Having done so now it appears that you are experiencing 2 problems, possibly the same bug, possibly not.

1. You are losing mail when you move it between folders (either manually or by filter). This might be:

http://bugzilla.gnome.org/show_bug.cgi?id=446276

2. You are sending mail but not getting a copy of your sent mail. This might be:

http://bugzilla.gnome.org/show_bug.cgi?id=436802

I too am seeing both problems, but I am not convinced they are the same bug, for the simple reason that I only experience the first one with my blueyonder account and the second one only with my webmail.us account. Also the second one is really hard to reproduce whereas the first one usually only takes a few tries.

Jerome, does any of the missing mail show up in your trash folder? Did you ever get a debug log with CAMEL_DEBUG=ALL for either of the IMAP servers you have used?

Losing data is a serious issue. Could someone please bump up the importance of this bug?

Hi Guys and Gals

I've recently come across this problem as well. 2 Things:

1. On the mail sending side:
Mail destined for the sent items folder on the IMAP server seem to be randomly dropped. I've tried as best I can to reproduce but no luck. Another thing I've noticed is that it doesn't happen just to one specific messages but rather to all messages sent within a certain time period, this time period is also completely random as far as I can see. Therefore I doubt this bug is triggered by the message being sent, seems to almost be a time triggered event.

2. On the mail moving front:
Same thing that Simon has logged, mails being moved either manually or by filters seem to be copied to the new location and then deleted from the original.

This is a huge problem for one of my users that receives roughly 500 messages on an average day, and sends out between 200 - 300 messages a day as well.

I agree with Simon that this bug needs to be bumped up the list of importance asap!

Changing the setting will not make the bug fixed faster, there is thousand of people using evolution and sending and receiving hundred of mails a week without any issue. The bug has no duplicate and very few subscriber, it's not easy to work on it for somebody not getting the bug without a way to trigger it. Maybe somebody getting the issue should send it upstream on bugzilla.gnome.org, the people writting the software might have a better idea about it

Sebastien Bacher wrote:
> Changing the setting will not make the bug fixed faster, there is
> thousand of people using evolution and sending and receiving hundred of
> mails a week without any issue. The bug has no duplicate and very few
> subscriber, it's not easy to work on it for somebody not getting the bug
> without a way to trigger it. Maybe somebody getting the issue should
> send it upstream on bugzilla.gnome.org, the people writting the software
> might have a better idea about it
>
The problem here is that it is fully reproducible, but only on Ubuntu. I
even personally use Debian (Etch and Sid), and the problem does not show
while accessing the same server(s).
So this seems to be Ubuntu-specific.

> So this seems to be Ubuntu-specific.

That's really unlikely, the Ubuntu package has almost no change and not one likely to create a such issue, do you use the exact same configuration and version in Ubuntu and Debian? Very few people get the bug so it's likely due to something in the configuration

I experienced the bug with moving mails ending up in the trash folder with my webmail.us account for the first time this morning. Previously I had only experienced that bug with my blueyonder account and was suspicious this was caused by a blueyonder server problem; I can rule this out now. log file attached.

I have been running Evo in debugging mode for several weeks now and have not experienced a repeat of the issue with copies of sent mails going missing. Very frustrating, particularly as another member of my family has experienced the exact same problem with Evolution and webmail.us on 3 occasions.

Will send log file to upstream bug report as well. Is someone going to link this bug report with either/both of the upstream bug reports I referred to in Comment #14?

Thank you for your work on that, I've added an upstream bug watch now

upstream argue that's a server bug and not an evolution one

Le jeudi 18 octobre 2007 à 13:50 +0000, Sebastien Bacher a écrit :
> upstream argue that's a server bug and not an evolution one

Read my report: it occurs on both Dovecot and Courier IMAP servers on
Debian Etch, and does not occur on the same servers with Evolution from
Debian Etch or Sid.

Honestly, I cannot do much more than report something reproducible,
point out as closely as I can the conditions I tried to pinpoint the
problem.
This is clearly a Ubuntu Evolution bug.

Now that Gutsy is out, we will probably upgrade and see what happens,
but I would have expected more attention for such a bad bug. It is
already too late for us to really care about this problem.
--
Jérôme Warnier
FLOSS Consultant
http://beeznest.net

Jerome, repeting that the bug is an ubuntu evolution one will not make the statement become true, the distribution has no specific code changes that could create that so if there is a bug that's an upstream one and will also happen on Debian or any other distribution using the same version. As written before it might come from a configuration difference. Anyway this bug is hard to trigger and nobody provided a reliable way to trigger it on a stock installation, it might well happen only using some buggy server as suggested by upstream which would explain because it happens only to some users. Did you read the upstream comments?

I can confirm this behaviour with gutsy. I am using debian/etch postfix as IMAP server. Moving mails (to another account) sometimes just flags them as deleted. As the mailserver is hosted in a vserver I assume that this might be related to http://bugzilla.gnome.org/show_bug.cgi?id=390740 (Evolution pegs cpu when connected to overloaded imap server).
Maybe the copy operation does not finish in time and just gets lost?

using CAMEL_DEBUG=all evolution >& evo.log I get these exceptions when moving three mails between imap accounts (i can undelete them and repeat the procedure ... it reliably fails ... does not peg the cpu):

(evolution:31022): camel-imap-provider-CRITICAL **: imap_command_start: assertion `store->ostream!=NULL' failed

(evolution:31022): camel-imap-provider-CRITICAL **: imap_command_start: assertion `store->ostream!=NULL' failed

(evolution:31022): camel-imap-provider-CRITICAL **: imap_command_start: assertion `store->ostream!=NULL' failed
removed 0
 Vfolder update
uids match:
Thread b42f1b90 >
CamelStore('imap://<email address hidden>/;imap_custom_headers;check_all;use_ssl=when-possible;command=ssh%20-C%20-l%20%25u%20%25h%20exec%20/usr/sbin/imapd'):get_folder('INBOX', 0) = 0x83f8e88
< b42f1b90 >
Thread b42f1b90 >
CamelStore('imap://<email address hidden>/;check_all;command=ssh%20-C%20-l%20%25u%20%25h%20exec%20/usr/sbin/imapd;use_ssl=when-possible;imap_custom_headers;override_namespace'):get_folder('INBOX', 0) = 0x85ce95c
< b42f1b90 >
Thread b42f1b90 >
CamelFolder:get_message('INBOX', '2640') =
class: CamelMimeMessage
mime-type: text/plain; charset=iso-8859-1
content class: CamelDataWrapper
content mime-type: text/plain; charset=iso-8859-1
< b42f1b90 >
Thread b42f1b90 >
CamelFolder:get_message('INBOX', '2655') =
class: CamelMimeMessage
mime-type: text/plain; charset=ISO-8859-1; format=flowed
content class: CamelDataWrapper
content mime-type: text/plain; charset=ISO-8859-1; format=flowed
< b42f1b90 >
Thread b42f1b90 >
CamelFolder:get_message('INBOX', '2639') =
class: CamelMimeMessage
mime-type: text/plain; charset=utf-8
content class: CamelDataWrapper
content mime-type: text/plain; charset=utf-8
< b42f1b90 >
 Searching for changed matches ' (and

       (match-all (< (get-received-date) 1136070000))

       (match-all (> (get-received-date) 1104447600))

  )
'
[... just vfolder updates ...]
(evolution:31022): camel-imap-provider-CRITICAL **: imap_command_start: assertion `store->ostream!=NULL' failed

(evolution:31022): camel-imap-provider-CRITICAL **: imap_command_start: assertion `store->ostream!=NULL' failed

(evolution:31022): camel-imap-provider-CRITICAL **: imap_command_start: assertion `store->ostream!=NULL' failed

I forgot to mention that the IMAP server is cyrus ... postfix is SMTP of course. That makes it three mailservers ...

In my case, the server is a powerful (recent 2 x dual-core Opteron with 4GB RAM) physical machine for about 50 users (mostly on Ubuntu using Evolution or some on Windows using Thunderbird) on the IMAP server. Really unlikely to be overloaded, and that's also what graphs (Munin) show.

We went to Gutsy lately (last week), and I've not heard happen it again.

Hope it helps.

THIS IS A CRITICAL BUG!!!!!!
I HAVE JUST LOST A BUNCH OF MESSAGES.
THIS IS A COMMON PROCEDURE THAT I HAVE DONE MANY TIMES WITH KMAIL AND THUNDERBIRD.
SWITCH TO EVOLUTION HAS BEEN FOR ME A VERY BAD IDEA.

I'm on Hardy, by the way.

My case was moving folders from the server to the local folders.

do you really think that using this tone is appropriate there?
you are talking to people which are mostly volunteers and try to fix your issues

You are right.
I'm sorry, I didn't men to be rude.
Having lost so much information made me angry, and when I see the importance set to Medium and the same bug upstream saying it's not a bug, just made me explode.
Sorry

I am experiencing this bug. It is a critical one to me too, because it can lead to the loss of many e-mails, and I have actually lost some before noticing there were still available when checking "show deleted messages".

I tell you how I can reproduce it on my machine :

I have a professional pop account, an imap account on Free.fr and I access my gmail account through imap.

I want to switch to gmail so I copied all my message from the local folders to gmail. It worked almost perfectly (except some subfolders thad failed to be created but I think it is gmail related).

Then I tried to copy my messages from the imap account to gmail : no mail were transfered. I then tried to move manually some mails from the imap Inbox to gmail : the messages simply disappeared and I had to check "show deleted messages" to recover them.

Please at least prevent users to do dangerous operations, mails are very important data.

I use an up to date Hardy 64bits.

I attach a log obtained by opening evolution with "CAMEL_DEBUG=all evolution >& evo.log", trying to move one message from the imap account to gmail and closing evolution. I have not been able to see anything suspicious but maybe you will.

Update : I have just tried all possible transfers with two messages (text I sent to myself and everything worked fine. I tried with another mail that wasn't from me and transfers from local to gmail and from imap to gmail failed. It seems to affect messages randomly, but an affected message would fail with a 100% probability.

It seems that mails with images are more likely to fail...

I hope I can help, thank you.

Jim.

May the people getting the issue comment on the upstream report? I'd close this report if the upstream report isn't re-opened in 2 weeks. setting the status as incomplete also.

I have the same/similar problem of losing e-mails in Evolution with IMAP:

If I send mails, they are quite often not put into the sent folder and just disappear though they reach their addressee. I have reported that:
http://bugzilla.gnome.org/show_bug.cgi?id=527071

Additionally, filters in Evolution are not working all the time, i.e. emails are not moved to a certain inbox folder. I think that's also because there's a IMAP problem with moving them.

Ben

Same problem here,
Patricio

I think that's another different issue not this one, since you sent it upstream I'm closing this one. thanks.

could somebody having the issue take that upstream on bugzilla.gnome.org?

no reply. closing the task.

The bug also happens here with multiple mailservers. Unfortunately it is hard to reproduce, but seems to happen more often if the connection to the mailserver is slow and there are several queued tasks.

Futhermore you normally find out about missing emails in the destination folder often days later.

I want to reopen this bug, because im having simulary issues on latest lucid (server and client)

Here is the software i use for this tests:
dovecot - Version: 1:1.2.9-1ubuntu5 (on server)
evolution - Version: 2.28.3 (client app)
thunderbird - Version: 3.0.4 (client app)
N900 Mailclient (don't have more info's)

Overview:
Moving folders inside an imap account with evolution. After move they are disapeared on all other mailclients (thunderbird, n900). In Evolution the Folder still exists and is accessable. On serverside (filesystem) the folder has been renamed (moved: example .test.move -> .move) but the folder is not seen on the other clients.

Moving the folder with thunderbird works as aspected.

Detail (this is what i do):
I create a new folder (test). This folder is then showing up in all mailclients.
I create a subfolder inside test (move). Newly created folders in evolution work as aspected.
I move the subfolder to root of mailaccount. The folder is moved inside evolution and on the filesystem (server) and it disappears on other mailclients in the folder test but it does not show up in root of mailaccount.

This does not need an high priority because there is an easy workaround:
Create a new folder at location where you want to move that folder. Select all mail in old folder an move the mail to new folder.

I would be glade to help while i can reproduce this bug and hope i can provide any information necessary to help solve this issue.

There was is a warning message at the bottom in evolution. Did not see it at first...

But this does not point me to any solution. Sorry for providing this only in german, but i could not find any logging in syslog or .evolution inside my home directory