trackerd always use 100% CPU

...in Ubuntu Gutsy RC.

Same problem each time I explore my documents with nautilus, even I do it one minute ago.

Same here.
I upgraded from feisty.
But the problem isn't always present, today was the first time I noticed it.
It was 3 hours (big lunch today) consuming 100% of one of my cpu cores (I have a core2duo).
I had to kill it.

can this problem be related to this one:

https://bugs.launchpad.net/ubuntu/+source/tracker/+bug/155244

can they be merged?

depends - does the disk space usage get ridiculous?

if so then its likely to be a dupe

if not it could be something else,

Here is my /home/casa/.cache/tracker:

total 218880
drwxr-xr-x 2 casa casa 4096 2007-10-28 15:35 .
drwxr-xr-x 3 casa casa 4096 2007-10-10 01:25 ..
-rw------- 1 casa casa 352256 2007-10-10 02:24 email-contents.db
-rw------- 1 casa casa 1648600 2007-10-28 15:35 email-index.db
-rw------- 1 casa casa 532480 2007-10-27 18:34 email-meta.db
-rw------- 1 casa casa 53391360 2007-10-28 15:35 file-contents.db
-rw------- 1 casa casa 69638744 2007-10-28 15:35 file-index.db
-rw------- 1 casa casa 97034240 2007-10-28 15:35 file-meta.db
-rw------- 1 casa casa 1270920 2007-10-28 15:35 file-update-index.db

It uses a lot of memory, but I can't say it is not normal.

I have the same problem with trackerd using 100% cpu when launched, even when the computer is not idle. And it continues this way for hours on end until I kill it. How do we stop this.

What does the below mean...

28 Oct 2007, 13:06:29:029 - ERROR: execution of prepared query UpdateFile failed due to constraint failed with return code 19
 theres a whole lot of them in ~/.local/share/tracker/tracker.log

sefs,

that is not good - can you email the log file (<email address hidden>)

it might be a harmless file move via a temp file and tracker stumbles over the race condition but I would need to see log

Also pls make use of throttle in tracker-preferences to throttle down cpu usage if it bothers use

You will need to restart trackerd for prefs to take effect

we have fixed corruption issues which might cause endless indexing in svn

Im not 100% sure this is fixed so when 0.6.4 hits repos pls retest and let me know if it persists

(make sure you do trackerd --reindex)

Hi the update hasn't seem to hit the gutsy repos, I've been watching for it. Does anyone know when it will arrive?

we are still fixing bugs for 0.6.4 - I cant promise when it will be released but early next week is looking good

This may be a "me too".

On a fresh install of gutsy/amd64 (2.6.22-14-generic), trackerd (0.6.3) uses outrageous amounts of CPU (currently > 1:30:00) and memory (currently ~1GiB). The trackerd database (assuming that's it in ~/.local/share/tracker/data/common.db) is currently of reasonable size.

Originally, I'd written "to index a near-empty home directory" above, but it appears that by default, trackerd is configured to index all mounted filesystems. IMO, it seems like a bad choice of default for an individual user's tracker to go trying to index everything on the system. I have ~800GiB of disk (removable HDDs for backup, video, etc.) currently mounted in addition to the normal stuff.

Screenshot of outrageous mem/CPU use attached.

Emmet,

the actual index is in ~/.cache/tracker

And we only index your home directory by default so unless you have mounts in your home it should not go and index anything else

The mem leaks and corruption issues have been fixed in svn

we have one serious bug to fix before releasing 0.6.4 and thats to stop indexing when disk space is low

Please give tracker another go when 0.6.4 has been released and backported into gutsy

I found the database in ~/.cache/tracker, came back here to report what I found, and saw your (Jamie's) message. Thank you for the helpful information.

Before the earlier posting, I had never opened the Tracker Preferences dialog in my life. I never even knew it was there. Under "Indexing" in the "Files" tab, "Index mounted directories" is checked.

Anyway, I totally take it back about the database size being reasonable! Holy cow! Look at these file sizes!

    file-index.db: 1432714192
    file-meta.db: 3284840448

That's 4.7GB!

I have a symlink in my new home directory to my old home directory mounted read-only. The total size of my old home directory is ~2.5GB. The total size of my new home directory is negligible (pretty fresh install).

The total size of the other mounted filesystems, however, is *not* negligible: about 44GB of archived stuff (multiple machines, some stuff up to 10 years old). I don't have my backup drive in right now (500GB) but I just mounted a 300GB drive with 161GB of stuff on it -- thankfully trackerd isn't running -- good ole killall.

Given the awful gaffe in Nautilus (it thinks the mounted partitions in the two sides of my mounted raid mirror are separate unmounted partitions), it's not impossible that tracker is trying to index all the mounted RAID-based partitions twice too.

I've never used it, so I won't miss it, so I just uninstalled it. Maybe I'll give it a whirl later as you suggest.

Emmet,

 thats the corruption bug - it causes endless indexing and ever growing index files

We have fixed that in svn.

We do not follow symlinks nor do we index stuff unless its mounted under home by default.

index size is almost always less than 25% of total text based stuff and non-text based stuff (like videos and music files) have very little effect on index size.

Jamie,

Thanks for that. I'm a mainly text-based user, so it might actually be pretty useful for me. I'm a dinosaur who finds a GUI handy mostly because it lets me have a half-dozen shells and text-editors open side-by-side. I've always found "grep -r" a wee bit slow. If tracker gets a nice CLI, I might use it quite a bit ;o)

I definitely didn't intentionally turn on the "index mounted directories" option, though. I would have had neither cause nor opportunity to enable an option that I didn't know existed for a program that I didn't know existed. That it should somehow have become enabled when it it should be disabled by default is more than a bit weird.

Emmet.

Emmet,

the index mounted directories option only applies to mounts in your home directory or any other path you have explicitly set in the watches/crawl directory prefs

It will *not* index stuff outside of this including system mounts in /media

Jamie,

It would simply never occur to me to mount something in a user's home directory. I therefore misunderstood "index mounted filesystems" to mean "index mounted filesystems" ;o)

Emmet.

2007/11/24, Emmet Caulfield <email address hidden>:
> Jamie,
>
> It would simply never occur to me to mount something in a user's home
> directory. I therefore misunderstood "index mounted filesystems" to mean
> "index mounted filesystems" ;o)
>

The wording in tracker-preferences is indeed a bit unclear about that
point. Do you have a good suggestion how this setting could be named
in t-p (best with less than 4-6 words)

Cheers,
Michael

--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

Possible pref naming suggestions:

"Follow mounted filesystems"
"Follow mounted filesystems in indexed locations"

Hi Michael,

My humourous (?) remark was intended to allude as much to my own fixed mindset about where things are mounted as to any ambiguity in the wording.

I must admit that it is tough to capture the intention precisely in a few words. I'm not sure I'd grok what "follow" means in Ron's suggestion if I saw it for the first time, though. How about "index filesystems mounted in indexed places"?

Emmet.

I get both issues appearing, 100% usage and massive index.

I mount nfs shares in my home dir ;-/ it's the only place many people are allowed to mount things on multiuser systems. However it seems that trackerd should default to not crossing filesystems by default.

-rw------- 1 aenertia aenertia 12K 2007-11-19 21:36 email-contents.db
-rw------- 1 aenertia aenertia 1.2M 2007-11-28 10:17 email-index.db
-rw------- 1 aenertia aenertia 96K 2007-11-19 21:37 email-meta.db
-rw------- 1 aenertia aenertia 449M 2007-11-29 00:03 file-contents.db
-rw------- 1 aenertia aenertia 258M 2007-11-28 22:51 file-index.db
-rw------- 1 aenertia aenertia 79M 2007-11-21 13:18 file-index-final
-rw------- 1 aenertia aenertia 6.7M 2007-11-21 22:36 file-index.tmp.1
-rw------- 1 aenertia aenertia 12M 2007-11-23 17:24 file-index.tmp.10
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:31 file-index.tmp.11
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:43 file-index.tmp.12
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:45 file-index.tmp.13
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:45 file-index.tmp.14
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:46 file-index.tmp.15
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:46 file-index.tmp.16
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:47 file-index.tmp.17
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:47 file-index.tmp.18
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:49 file-index.tmp.19
-rw------- 1 aenertia aenertia 14M 2007-11-21 22:56 file-index.tmp.2
-rw------- 1 aenertia aenertia 13M 2007-11-23 18:00 file-index.tmp.20
-rw------- 1 aenertia aenertia 13M 2007-11-23 18:11 file-index.tmp.21
-rw------- 1 aenertia aenertia 13M 2007-11-23 18:32 file-index.tmp.22
-rw------- 1 aenertia aenertia 16M 2007-11-27 11:07 file-index.tmp.23
-rw------- 1 aenertia aenertia 15M 2007-11-27 14:50 file-index.tmp.24
-rw------- 1 aenertia aenertia 16M 2007-11-27 17:16 file-index.tmp.25
-rw------- 1 aenertia aenertia 16M 2007-11-28 12:21 file-index.tmp.26
-rw------- 1 aenertia aenertia 16M 2007-11-28 22:11 file-index.tmp.27
-rw------- 1 aenertia aenertia 16M 2007-11-21 23:22 file-index.tmp.3
-rw------- 1 aenertia aenertia 14M 2007-11-22 00:07 file-index.tmp.4
-rw------- 1 aenertia aenertia 14M 2007-11-23 16:51 file-index.tmp.5
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:01 file-index.tmp.6
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:05 file-index.tmp.7
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:15 file-index.tmp.8
-rw------- 1 aenertia aenertia 13M 2007-11-23 17:19 file-index.tmp.9
-rw------- 1 aenertia aenertia 9.3G 2007-11-29 00:32 file-meta.db <---------------- And rising ;-)
-rw------- 1 aenertia aenertia 32K 2007-11-29 00:32 file-meta.db-journal
-rw------- 1 aenertia aenertia 1.3M 2007-11-28 22:11 file-update-index.d

I have not touched ANY configuration for this program. Just standard gutsy install.

I upgraded to Ubuntu 7.10 and noticed that trackerd was consuming inordinate amounts of CPU. Reminds me of the same problem I had with the Beagle search thing. The combination of consuming all CPU and not being able to find *anything* in the tracker search tool means I'll be uninstalling it, but I wanted to leave feedback regarding this being a problem. I'm running 0.6.3. I haven't modified any config for this, it just appeared after upgrading.

Yesterday, I think it used 58 minutes of CPU time. This morning it did eventually stop about a half hour after boot; whereas, I had to kill it yesterday.

I suppose it will appear again when I upgrade to 8.04, and maybe it will actually allow me to search for something.

The tracker in the repos still sits at 0.63. What became of the update to stop the 100% cpu usage.

0.6.4 is now out - awaiting packaging and backports to gutsy

I have the same issue, but i've noticed that when you delete your .cache directory and let tracker to scan dirs once again, it work for some time ;-) but mostly it isn;t
There is still 0.6.3 in Gutsy repos is it going to change?

Closing as tracker 0.6.4 pauses itself when there's disk IO from other files.

There is no chance of getting this into gutsy-updates, as we can't test it properly that there's no regressions, but we certainly can get it (and I certainly would like to) in gutsy-backports. So please, test it from https://launchpad.net/~pochu/+archive and send feedback to bug 175676.

Cheers

I had many ~/.cache/tracker/*tmp* too.
See my solution at https://bugs.launchpad.net/ubuntu/+source/tracker/+bug/155244/comments/15

One more to add to this thread. I've been running a clean Gutsy install on my core2 duo machine. Tracker 0.6.6 came pre-installed, and I noticed that several minutes after startup (and I think sometimes after enough idle time) trackerd begins maxing out one of the cores. If I leave it alone for long enough, 5 minutes or so, it eventually calms down, but it is a bit distressing to see (and on my previous single-core system, it would cause a complete lock-up). I tried looking into configuration options and configured trackerd as in the attached file. basically, I tried to throttle back every aspect of the indexer as far as possible.

The new configuration seemed to have no noticeable effect on trackerd's habits, nor does the "Pause All Indexing" option in the system tray dropdown menu (it appears as though the setting takes effect only after the period of heavy cpu load ends). Memory usage appears contained at only 12 MB or so. My cache directory looks commensurate to cuby's in terms of file sizes.

Tracker seems like a very useful app, and I'd love to use it, but sometimes I feel compelled to kill it, and therefore I can't rely on its indexes being up to date.

Cheers,
Matt