tracker doesn't leave any for the rest of the class

Bug #158040 reported by Scott James Remnant (Canonical)
10
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Won't Fix
Medium
Unassigned
linux-source-2.6.22 (Ubuntu)
Won't Fix
Undecided
Unassigned
tracker (Ubuntu)
Fix Released
High
Unassigned

Bug Description

Binary package hint: tracker

inotify sucks.

In order to watch a directory tree, you have to walk the tree and add an inotify watch for every directory in the tree; watching the root won't tell you about changes in children directories.

To make matters worse, there is a limit on the number of inotify watches you can have -- and that limit is per-user rather than per-instance. (If it were per-instance, you could just add another inotify instance -- but then there are also per-user limits on the number of inotify instances, but I'm digressing).

I have what I think is a pretty typical developer's home directory, it's certainly not that old so hasn't had time to grow ridiculously large yet.

It has over 16,500 directories in it. Most of these seem to be under .svn, .bzr and .git -- I guess they like directories.

This is over twice the number of per-user inotify watches available.

trackerd happily adds watches until it runs out -- denying any other user process from using inotify at all!

This is not ideal.

Changed in tracker:
importance: Undecided → High
Revision history for this message
TJ (tj) wrote :

Confirmed here too. I spent a few hours trying to solve an issue whereby one of my development projects that uses inotify couldn't add new watches. I checked the maximum limits but couldn't find a way to get inotify to report a list of the watches so in the end gave up on it.

Like Scott I have a *lot* of directories configured in two trees (Projects/ and SourceCode)/. SourceCode has copies of the source of about a hundred packages I have debugged or inspected and in addition the complete kernel, l-u-m and l-r-m trees for Feisty, Gutsy and kernel.org.

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

The quick fix is to increase number of inotify watch limit by default to 64,000 or something high

this should be a bug against the kernel or boot config though rather than tracker

short term fix :
echo 65565 > /proc/sys/fs/inotify/max_user_watches

long term: apply to a boot up script

For hardy, I will attempt to write a kernel patch to generate a persistent ring buffer log file of all file changes which can be used by user space apps instead of watches - this would dramatically increase startup as no crawling of directories to place watches on them would be needed

Changed in tracker:
status: New → Invalid
Revision history for this message
Scott James Remnant (Canonical) (canonical-scott) wrote :

Jamie: this isn't an invalid bug -- as you point out, it's either a bug in tracker for blithely consuming all instances, or a kernel bug for inotify being crap in the first place.

Since you are working on a kernel patch, I've opened a task for that and assigned it to you

Changed in linux-source-2.6.22:
assignee: nobody → jamiemcc
Changed in linux-source-2.6.22:
assignee: jamiemcc → jamiemcc-blueyonder
Revision history for this message
Martin (martin615) wrote :

Jamie,

That sounds great! I've been pretty peeved about inotify:s lack of built-in recursive directory monitoring. A ring buffer of all file changes would solve that one automatically... and the max_user_watches problem to boot. :)

Only hope you can get it accepted upstream...

("this would dramatically increase startup"... I assume you meant to say decrease :)

Revision history for this message
Stefan Tzeggai (alfonx-deactivatedaccount) wrote :

Workaround instructions to put it in a bootscript. Place
  fs.inotify.max_user_watches=20000
in /etc/sysctl.conf

Just For Info

Changed in linux:
assignee: nobody → ubuntu-kernel-team
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
Ben Collins (ben-collins) wrote :

The ring-buffer approach is not going to be accepted. First off, a list of files that have changed is useless. The full path does not accomodate symlinks or hardlinks. Secondly, we don't want users being able to see all the files that have changed. That's a data leak that exposes security concerns.

Closing kernel task, as there seems to be no real solution for the kernel. Using sysctl to increase the default (since the default is supposed to be safe) seems like the only sane idea.

Changed in linux:
assignee: ubuntu-kernel-team → nobody
status: Triaged → Won't Fix
Revision history for this message
Ben Collins (ben-collins) wrote :

Unlikely a fix would be applied to 2.6.22/gutsy even if one was developed.

Changed in linux-source-2.6.22:
assignee: jamiemcc-blueyonder → nobody
status: New → Won't Fix
Revision history for this message
Scott James Remnant (Canonical) (canonical-scott) wrote :

If the kernel won't fix it, then this is a bug in tracker

Changed in tracker:
status: Invalid → Confirmed
Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

tracker (0.6.4) leaves 500 inotify watches free - is that enough?

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

marking as fixed in 0.6.4

Changed in tracker:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.