Thunderbird creates a lot of "tmprules-NNN.dat" files in /tmp

That file is created in nsMsgFilterService::SaveFilterList(), and there's must have been something gone wrong, because the tmpFiltersFile->Delete() is never reached. Can you check if there is a file in your profile called tmprules.dat or msgFilterRules.dat, which is read only ?

The are all rw.
> find . -name msgFilterRules.dat -ls
4263055 4 -rw------- 1 username staff 25 Jan 7 02:55 ./Mail/Local Folders/msgFilterRules.dat
966652 4 -rw------- 1 username staff 25 Jan 7 02:55 ./Mail/News & Blogs/msgFilterRules.dat
4263270 8 -rw------- 1 username staff 3353 Mar 25 18:15 ./Mail/pop.gmail.com/msgFilterRules.dat

Happening under WinXP as well (TB2 build 20070326).

Shouldn't a temporary file be deleted anyway, no matter what errors occur in the function creating it?

I observe this on fc5 (2.6.18.8) with version 2.0.0.0 (20070326)

Bug confirmed under Thunderbird 2.0.0.0 (20070419) (Gentoo build, amd64).

I've been getting these folders appearing in my News and Blogs for some time, and just in the last day or two in the Local Folders section too. I'd assumed they were created because we use two different versions of Thunderbird (on two computers) to access our files across a network, but presumably this is not in fact the cause?

PS I'm using Thunderbird Trunk Alpha (nightly), and my wife uses the release version (not Alpha or Beta).

Bug confirmed under Thunderbird 2.0.0.0 (20070518) (Gentoo build, x86).

I have it too.

Thunderbird: version 2.0.0.0 (20070519)
Gentoo x68

I use thunderbird to connect to an IMAP server, and I get these files in /tmp.

Ubuntu Feisty Thunderbird 2.0.0.0 20070326 also getting this problem.

Also on Solaris/SPARC with IMAP account(s) - I've been seeing this for a while now and finally decided to search the bugs DB.

As a w/a, I'm using a cron job deleting the files.

I discovered that, despite these not being visible in the Folders list in Thunderbird, it is also creating loads of these files for each of my 10 or so different email accounts!

I've never heard of a cron job, but Googling it looks like it's a Unix thing. Some of us still use a Microsoft operating system even if our other software is Open Source ;-)

I can confirm this bug under Thunderbird 2.0.0.0 (Fedora 7 package: thunderbird-2.0.0.0-1.fc7).

I see a new file in /tmp/ created for each time thunderbird checks for new messages. First file is /tmp/tmprules.dat, then /tmp/tmprules-1.dat, then /tmp/tmprules-2.dat, and so on.

If I remove all message filters the problem remains.

I believe the problem is part of the Junk Mail system. I can cause the tmprules file to be created when:

1.) I have (at least) 2 accounts
2.) One account has Junk Settings->Trust junk mail headers set by: [Spamassassin]
3.) The other account has Junk Settings->Enable adaptive junk mail controls for this account or Junk Settings->Do not mark mail as junk if the sender is in [Personal Address Book].

If I only have one account, enabling the junk mail settings doesn't cause new tmprules.dat files to be created.

(In reply to comment #13)
> I can confirm this bug under Thunderbird 2.0.0.0 (Fedora 7 package:
> thunderbird-2.0.0.0-1.fc7).

Just upgraded to Thunderbird version 2.0.0.4 (20070615), it still has the same incorrect behavior.

(In reply to comment #8)
> Bug confirmed under Thunderbird 2.0.0.0 (20070518) (Gentoo build, x86).
>

I've now upgraded to Thunderbird 2.0.0.4 (20070618) (Gentoo build, x86) and the issue is still here. I'm checking exclusively IMAP accounts (2 with SSL, 3 without), and these files appear in /tmp/ (also seeing left-over files in my profile as per Jo Hermans' post and Pawel Worach confirmation).

Created attachment 270310
Remove tmprules

I think tmpFiltersFile should be always removed before SaveFilterList() finished.

Alexey, have you been able to reproduce the problem? I think your fix is probably the right thing, but I'd really like to know what's making things go wrong in the first place.

I have reproduced this problem with 2.0.0.0 and 2.0.0.4 versions. I will debug SaveFilterList() to answer your question.

oh, thank you! getnewmail shouldn't cause SaveFilterList to get called in the first place - so a stack trace would be interesting.

I can also confirm this bug on seamonkey 1.1.2.

Sorry for the long silence.
I put into this SaveFilterList() function some debug messages (ZZZ prefix):

...
ZZZ nsMsgFilterService::SaveFilterList() InFunction()
ZZZ nsMsgFilterService::SaveFilterList(): filterFile /usr/lib/thunderbird/isp/SpamAssassin.sfd
ZZZ nsMsgFilterService::SaveFilterList(): ret = filterList->SaveToFile(tmpFileStream): NS_SUCCEEDED
ZZZ nsMsgFilterService::SaveFilterList(): ret = tmpFiltersFile->CopyToDir(parentDir): NS_FIALED
###!!! ASSERTION: error opening/saving filter list: 'NS_SUCCEEDED(ret)', file nsMsgFilterService.cpp, line 215
Break: at file nsMsgFilterService.cpp, line 215
Begin mail message delivery.
Abort mail message delivery.
Opening file SpamAssassin.sfd failed
...

I think, it's clear enough. /usr/lib/thunderbird/isp/SpamAssassin.sfd is not writable by user in Linux.

Created attachment 272146
Remove tmprules

Bug assigned to Nobody.
How can review this patch ?

Comment on attachment 272146
Remove tmprules

thx, Alexy, that makes a lot of sense, as to why only some people would have this issue. And your fix looks fine - I'm not sure you need to check if the file exists before calling delete but it shouldn't hurt.

Can you give me a stack track of when we get into this scenario? We shouldn't be trying to save over the .sfd file at all...

I use an executable built with that latest patch above. The spurious tmprules-*.dat files are gone, but the Junk Settings->Trust junk mail headers set by: [Spamassassin] option still works unrealiable - for no obvious reasons, about half of the SPAM-mails clearly marked as SPAM by Spamassassin are not treated as SPAM by thunderbird.

(In reply to comment #24)
> I use an executable built with that latest patch above. The spurious
> tmprules-*.dat files are gone, but the Junk Settings->Trust junk mail headers
> set by: [Spamassassin] option still works unrealiable - for no obvious reasons,
> about half of the SPAM-mails clearly marked as SPAM by Spamassassin are not
> treated as SPAM by thunderbird.
>

I have the same problem that half of the emails marked as SPAM by Spamassassin are not detected. This problem (and the tmprules-*.dat files, of which on my system thousands are generated each day due to my high mail volume) has been there since I switched to 2.0

Users of the icedove Debian package also <a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=440698#5">reported this bug</a>.

As an icedove user, turning off the "trust spamassassin" option, that was on on one of the accounts, fixed it for me immediately. /tmp was littered with thousands of those files (its only cleared on reboot I guess).

peter

Hello,

This bug happens as confirmed in Thunderbird 2.0.0.0 to 2.0.0.6 under Windows XP. We have detected the creation of tmp files because the computer is VERY SLOW while using Thunderbird.

First of all, our users told us that "when creating an email, Thunderbird doesn't responds to the keyboard" and after some seconds "the letters and words appears suddenly". Thunderbird is slow (seems like is hanging for some seconds) while using menus, dialog boxes or simply trying to close it. When this happens the use of CPU of Thunderbird increases up to 99% (as showing in Task Manager).

We've used FileMon/RegMon for watching what's Thunderbird doing while is "not responsive" and we've seen that Thunderbird is creating a lot of "tmprules*.dat" files in %TEMP% folder of the user (i.e. C:\Documents and Settings\<user>\Temp). I've seen the creation of thousands of files in just some 4-5 seconds. The temporary folder of the users have tmprules-1.dat up to tmprules-9999.dat files.

I suppose that the CPU hung of 99% is due to the fact that some Thunderbird routine is creating these files and doesn't let Windows react to user inputs (mouse, keyboard, screen, etc.).

We suppose too that these problem appears every time that Thunderbird checks the inbox for new messages and this problem appears if you have message filters but too if you don't have these filters.

We have SOLVED this problem (the slowdown of the PCs NOT the creation of tmp files) doing the following:

- Closing Thunderbird
- Deleting msgfilterrules.dat of every account in the user profile
- Deleting tmprules*.dat in the temp folder of the user

Then, Thunderbird is VERY responsive. It doesn't hangs but the temporary files continue appearing in the temp folder (at a 2-3 files per session rythm not thousands of them). We don't know if the problem will appear in the futur (for example when the file number will be 9999) so we're watching for a solution in the code of Thunderbird.

We will look for the settings related to SpamAssassin but I think (I need to check this in an user computer) we have them disabled.

I hope this information will point the route to a solution.

Thank you.

PS: Sorry for my poor english :-(

Hi again,

I've forget saying that we've started using FileMon/RegMon for monitoring Thunderbird when an user asked us "why their message filter rules dissapear after closing TH". We monitored their profile directory to see if the msgFilterRules.dat was updated and we saw the creation of thousands of tmp files.

PS: Remember this after reading https://bugzilla.mozilla.org/show_bug.cgi?id=390992 ;-)

Here are our Account Settings > user@account > Junk Settings

[ ] Enable adaptative junk mail controls for this account
[X] Trust junk mail headers set by: SpamAssassin

So, the trust junk is checked but the adaptative junk is not. Is Thunderbird doing something with this configuration? Are these two independent options? Or the trust option only works when the first one is enabled?

Thanks.

If you manually change /usr/lib/thunderbird/isp/SpamAssassin.sfd so that it's writeable by the user, does that fix the problem?

I vaguely remember there being an issue about where we look for the .sfd files as well, but I can't find that bug.

I'm still getting this problem even after I turned off the Trust Spamassassin thing. I don't know when the files are generated, but it is conceivable that they appear when my wife is using Thunderbird 2.0.0.0 (20070326)through the network, or it could be when I am using Thunderbird 3.0a1pre (trunk nightlies)

looking again at this patch, it's not right for the trunk because of the conversion from nsIFileSpec to nsIFile. I can try to come up with a trunk version.

I think the patch would be ok for the 2.0 branch, however.

We don't know why this problem starts.

We only know that if this problem appears (slow pc caused by thunderbird blocked while creating tmp files) we can solve it by:

- closing thunderbird
- deleting tmprules*.dat and msgfilterrules.dat
- starting thunderbird

After taht, thunderbird works ok, but it starts creating again temporary files (this time, only a couple of files). We have seen that, when the users says to us "my thunderbird is slow", the problem is caused by the temporary files problem. We see in the %TEMP% folder up to 9999 temporary dat files.

We can't reproduce the problem after the solve it (by the way, we don't know if the problem will appear again if the count for the temporary files will be again 9999).

File SpamAssassin.sfd under Windows is under the program files directory so no writable by a "user".

Will all these files be created at once while running one getNewMail or step by step? If it's on one step do we have a problem in creating the file or why such a huge amount of files are created? The attached patch only handles the deleting of such temporary files.

Hi,

As we've seen while running FileMon, these files are created at once. Every check mail time thousands of these files are created. I don't know when/why the 9999 limit is reached. When we have detected the problem we saw this amount of temporary files.

At this moment we don't have any client with the slow problem after we did the deletion of the temporary files but ... we don't know if the problem will appear again when some magic number of temporary files is reached ;-)

We can't test the patch because we don't compile the Windows code by ourselves. We install/upgrade using the standard binaries.

Hello again,

The "slow problem" has appeared again.

Now we have debugged it and we have the FileMon (FileMonitor) log files where we can see what's doing Thunderbird when it works ok and when it works bad.

When working bad (fragments):
=============================

16 11:26:12 thunderbird.exe:2372 OPEN C:\DOCUME~1\CASTELL\TEMP\tmprules.dat SUCCESS Options: Open Access: All
17 11:26:12 thunderbird.exe:2372 QUERY INFORMATION C:\DOCUME~1\CASTELL\TEMP\tmprules.dat SUCCESS Attributes: A
18 11:26:12 thunderbird.exe:2372 CLOSE C:\DOCUME~1\CASTELL\TEMP\tmprules.dat SUCCESS
19 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1 NAME COLLISION Options: Create Directory Access: All
20 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1\CASTELL NAME COLLISION Options: Create Directory Access: All
21 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1\CASTELL\TEMP NAME COLLISION Options: Create Directory Access: All
22 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1\CASTELL\TEMP\tmprules.dat NAME COLLISION Options: Create Access: All
23 11:26:12 thunderbird.exe:2372 OPEN C:\DOCUME~1\CASTELL\TEMP\tmprules-1.dat SUCCESS Options: Open Access: All
24 11:26:12 thunderbird.exe:2372 QUERY INFORMATION C:\DOCUME~1\CASTELL\TEMP\tmprules-1.dat SUCCESS Attributes: A
25 11:26:12 thunderbird.exe:2372 CLOSE C:\DOCUME~1\CASTELL\TEMP\tmprules-1.dat SUCCESS
26 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1 NAME COLLISION Options: Create Directory Access: All
27 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1\CASTELL NAME COLLISION Options: Create Directory Access: All
28 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1\CASTELL\TEMP NAME COLLISION Options: Create Directory Access: All
29 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1\CASTELL\TEMP\tmprules-1.dat NAME COLLISION Options: Create Access: All
30 11:26:12 thunderbird.exe:2372 OPEN C:\DOCUME~1\CASTELL\TEMP\tmprules-2.dat SUCCESS Options: Open Access: All
31 11:26:12 thunderbird.exe:2372 QUERY INFORMATION C:\DOCUME~1\CASTELL\TEMP\tmprules-2.dat SUCCESS Attributes: A
32 11:26:12 thunderbird.exe:2372 CLOSE C:\DOCUME~1\CASTELL\TEMP\tmprules-2.dat SUCCESS
33 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1 NAME COLLISION Options: Create Directory Access: All
34 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1\CASTELL NAME COLLISION Options: Create Directory Access: All
35 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1\CASTELL\TEMP NAME COLLISION Options: Create Directory Access: All
36 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1\CASTELL\TEMP\tmprules-2.dat NAME COLLISION Options: Create Access: All
37 11:26:12 thunderbird.exe:2372 OPEN C:\DOCUME~1\CASTELL\TEMP\tmprules-3.dat SUCCESS Options: Open Access: All
38 11:26:12 thunderbird.exe:2372 QUERY INFORMATION C:\DOCUME~1\CASTELL\TEMP\tmprules-3.dat SUCCESS Attributes: A
39 11:26:12 thunderbird.exe:2372 CLOSE C:\DOCUME~1\CASTELL\TEMP\tmprules-3.dat SUCCESS
40 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1 NAME COLLISION Options: Create Directory Access: All
41 11:26:12 thunderbird.exe:2372 CREATE C:\DOCUME~1\CASTELL NAME COLLISION Options: Create Directory Access: All
42 11:26:12 thunderbird.exe:2372 CREATE C:\DOCU...

Created attachment 282995
Thunderbird use of tmprules.dat when is working properly

You can browse the log in a more efficient way using FileMon (from SysInternals/Microsoft).

Created attachment 282996
Thunderbird use of tmprules.dat when is working bad

You can browse the log in a more efficient way using FileMon (from SysInternals/Microsoft).

I have to ask: What is the purpose of creating tmprules.dat if we immediately go and delete it? I'm assuming that it is used to transfer data from one part of the program to another. Is there a better way of doing this that doesn't involve writing a temporary file?

Adding myself to Cc list. Reproduced on 2.0.0.6 on XP SP2 with an IMAP account. Filters disappear and new filters are not saved (only in the session), which is observed when ultimately 9999 files have been created in the users's temp dir.
For what it's worth, it only happens for users with non-admin rights. For admin users the issue does not happen. Could this be related to how the file is supposed to be deleted?

So, looking at comment 23 and comment 33. David, should we just land this on branch?

On trunk, move is used instead of copy, so there should be no need to delete the tmp file.

http://lxr.mozilla.org/mozilla1.8/source/mailnews/base/search/src/nsMsgFilterService.cpp#183
http://lxr.mozilla.org/seamonkey/source/mailnews/base/search/src/nsMsgFilterService.cpp#189

Comment on attachment 272146
Remove tmprules

requesting 2.0.x approval

the fix of this bug on branch 2.0 will also fix bug 403907

I think I was wrong in comment 42, trunk too should probably need to delete the tmp file when/if move fails. (But the patch should still land on branch.)

Can someone give me steps to reproduce this? Enable trust spam assassin and make sure program installation dir is not writable, then what? Would like to make sure what I had in mind works...

So *that's* where all those files are coming from and *that's* why filters are broken. I think this is a serious bug.. voting..

I can confirm it's still happening with 2.0.0.9.

Comment on attachment 272146
Remove tmprules

approved for 1.8.1.12, a=dveditz for release-drivers

MOZILLA_1_8_BRANCH:

Checking in mailnews/base/search/src/nsMsgFilterService.cpp;
/cvsroot/mozilla/mailnews/base/search/src/nsMsgFilterService.cpp,v <-- nsMsgFilterService.cpp
new revision: 1.48.2.5; previous revision: 1.48.2.4
done

Still open for trunk patch.

I will port this patch on trunk in few days.

Alas, while the applied patch does prevent the stray ".tmp" files from being created, it does not fix the problem that SpamAssassin headers are ignored, even if configured otherwise. This is a major problem for all those terrorized by lots of "inline-image-SPAM" which seems not to be detected by the Mozilla-internal junk mail code.

Does anybody have an idea how these two symptoms relate - as they occured first (seemingly) at the same revision level?

@Peter: For tje problem with SpamAssassin maybe you could open a new bug (or there is maybe already a dupe ...)
As if the stray tmp file is fixed, this bug will be RESOLVED-FIXED after verification of fix on trunk ...

Have a look to this bug: Bug 381589 "Trust Spamassassin option is not working in Thunderbird 2.0"

Bug fix verified on build Thunderbird/2.0.0.12pre ID:2008011303

As per comment 52 => verfied1.8.1.12

Created attachment 297462
Remove tmprules (patch for trunk)

Sorry for a delay.

Checking in mailnews/base/search/src/nsMsgFilterService.cpp;
/cvsroot/mozilla/mailnews/base/search/src/nsMsgFilterService.cpp,v <-- nsMsgFilterService.cpp
new revision: 1.63; previous revision: 1.62
done

Since yesterday's trunk build, a user who had this problem originally, msgFilterRules.dat has been deleted. After restore, the file has been deleted again.
Using IMAP, non-admin XP SP2.

(In reply to comment #56)
> Since yesterday's trunk build, (snip) msgFilterRules.dat has been deleted.
It's Bug 413680.

Either this bug is now not fixed or something very similar is happening. My message filters have suddenly stopped working. New filters cannot be saved and are eliminated when TB closes.

I have looked hard for the tmprules-xx.dat files, but have not been able to find more than a few stray ones

I am using 2.0.0.16, WinXP

(In reply to comment #58)
> I have looked hard for the tmprules-xx.dat files, but have not been able to
> find more than a few stray ones

To Robin Gaster:
Did you search for tmprules.dat too? ( for xx=0, tmprules.dat instead of tmprules-0.dat ) Did you search for tmprules-xx.dat/tmprules.dat at where? %temp% directory only? Did you search mail directory in which msgFilterRules.dat is held?
In a forum in Japan, following phenomena was reported.
 If garbage of ...\tmprules.dat exists("...\" is path of ...\msgFilterRules.dat)
 (a)msgFilterRules.dat can't be updated.(Exception is reported to Error Console)
    So all added message filters disappears after restart.
 (b)If msgFilterRules.dat doesn't exist, null msgFilterRules.dat is created.
    After null msgFilterRules.dat creation, problem of (a) occurs.
    So all created message filters disappears after restart.

yes, I searched for all versions of tmprules*.dta