Too many files opened at once

Created an attachment (id=1476)
The patch mentionned in the bug report

Created an attachment (id=1478)
The patch mentionned in the bug report

The previous patch was buggy (it destroyed the file
instead of appending to them).

Is this fix worth adding? Trade off cpu load vs. having a large number of files
open?

There's another possibility to raise the number of allowed open files per
process for the desktop user,
adding a line to /etc/security/limits.conf:

<username> hard nofile 2048

maybe this can be generalized to

@users hard nofile 2048

as a default.

I was aware of this configuration option, but I have found several
times torrents with 5000+ files and I don't really like having the
maximum number of file descriptor by process set this high.

Besides, when opening such a large torrent, the application seems
frozen. It is caused by the fact that opening the last file will
raise an exception, that is caught. In response to this exception
the program will try to read the glade description of the error
dialog box from the disk, which will result in another exception
being raised. That second exception is not caught and then the
GUI thread is dead ...

My patch was a first attempt at correcting this problem. Another
approach could be to read the maximum number of file descriptor
authorized for this process and only close some of them when there
is less than ten (or some other value large enough to be able to
load glade description file) remaining. The only problem is to
get the number of opened file descriptor currently in a portable
way ...

(In reply to comment #4)
> Besides, when opening such a large torrent, the application seems
> frozen. It is caused by the fact that opening the last file will
> raise an exception, that is caught. In response to this exception
> the program will try to read the glade description of the error
> dialog box from the disk, which will result in another exception
> being raised. That second exception is not caught and then the
> GUI thread is dead ...

At least, it is the case when the application is gnome-btdownload
(which seems to be the "official" bittorrent client for hoary). I
haven't tested with other graphical front-end.

nofiles = len(os.listdir('/proc/%d/fd/*' % getpid()))

that may be portable enough for Linux, but I'm not aware of another method.

The correct way to determine this value is to use sysconf. I am not sure
whether there is a Python API for it, and at any rate, the Python program cannot
know how many file descriptors are in use by the interpreter.

At any rate, this bug isn't serious enough to warrant an invasive change so
close to release, so let's revisit this after the Hoary release.

os.sysconf('SC_OPEN_MAX') gets you the value of the maximum allowed files. The
limits approach was to limit this change to the processes the user starts.
Another possibilty would be to keep the soft limit, but increase the hard limit,
so this value can be raised by the user by resource.setrlimit(...)

IMO it doesn't make sense if the interpreter tracks a record of all open files,
the number of files opened by extension modules and external libraries is
unknown to the interpreter and such tracking would be unreliable.

I have checked the Bittornado code, and they let the user specify
the maximum number of file that can be opened at once by a single
Storage object. If after opening a file this limit is reached the
last recently opened file is closed.

This is better than my approach of always closing all the file,
is more tested and more portable than trying to guess how many
open file-descriptor the python interpreter has got.

And it should be relatively easy to port it from Bittornado to
Bittorrent if it is the choosen solution.

I where just going to write a new bugreport about this issue but i found this one, just want to ping and say that this is still an issue in Edgy.

If i install bittorrent 5.0 (bittorrent_5.0.0_python2.4.deb) from www.bittorrent.com this problem is fixed.Why don't we just upgrade bittorrent 3.x to 5.x for Feisty?

Is this still an issue after 3 years of the original report?

This is still an issue on Gutsy, not sure about Hardy.

Thank you for taking the time to report this bug and helping to make Ubuntu better. You reported this bug a while ago and there hasn't been any activity in it recently. We were wondering is this still an issue for you? Can you try with latest Ubuntu release? Thanks in advance.

I tested the btdownloadcurses from the bittorrent package from Ubuntu 8.04, and it still does exhibit this bug. But is really outdated: version available in Ubuntu is 3.4.2-11ubuntu5, when upstream version is 5.0.

Moreover, the default bittorrent client in Ubuntu 8.04 is Transmission, and it doesn't have this bug. So I think that the bittorrent package should be deprecated, or even removed (the bittornado package give a completely compatible interface from user space and does not exhibit this error).

I would like to see an official bittorrent client in Ubuntu repos, but
I would also like to see it updated. Any chance of this happening? I
like to have the reference implementation available for testing
purposes...

On 9/25/08, Sylvain Defresne <email address hidden> wrote:
> I tested the btdownloadcurses from the bittorrent package from Ubuntu
> 8.04, and it still does exhibit this bug. But is really outdated:
> version available in Ubuntu is 3.4.2-11ubuntu5, when upstream version is
> 5.0.
>
> Moreover, the default bittorrent client in Ubuntu 8.04 is Transmission,
> and it doesn't have this bug. So I think that the bittorrent package
> should be deprecated, or even removed (the bittornado package give a
> completely compatible interface from user space and does not exhibit
> this error).
>
> --
> Too many files opened at once
> https://bugs.launchpad.net/bugs/13371
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in "bittorrent" source package in Ubuntu: Incomplete
>
> Bug description:
> The bittorrent client opens all the file at the beginning
> of the transfer, and thus can reach the configured max
> open file descriptor by process when trying to download
> large torrents.
>
> It is then completely frozen since a second IOError is
> raised and uncaught when the error dialog glade XML
> file is opened.
>
> The following patch fixes the problem by preventing
> bittorrent from opening too many file at once (but it
> is a bit more cpu / disk hungry since the files are
> opened and closed more often) ...
>

@drink: you have 'transmission' as default bittorrent client (and also updated).

While I appreciate transmission (simple, works) I shouldn't have to
point out that "bittorrent" is the reference client for the bittorrent
protocol/system/networks, not transmission. In theory, it should work
no differently - which is why it's nice to have around to compare to.

Nothing is stopping me from installing bittorrent manually (and I have
done so previously) but I would prefer to have the latest in the
repository.

Is there any reason the official client shouldn't be in the repo? I
agree it doesn't need to be on the CD.

On 9/27/08, Pablo Castellano <email address hidden> wrote:
> @drink: you have 'transmission' as default bittorrent client (and also
> updated).
>
> --
> Too many files opened at once
> https://bugs.launchpad.net/bugs/13371
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in "bittorrent" source package in Ubuntu: Incomplete
>
> Bug description:
> The bittorrent client opens all the file at the beginning
> of the transfer, and thus can reach the configured max
> open file descriptor by process when trying to download
> large torrents.
>
> It is then completely frozen since a second IOError is
> raised and uncaught when the error dialog glade XML
> file is opened.
>
> The following patch fixes the problem by preventing
> bittorrent from opening too many file at once (but it
> is a bit more cpu / disk hungry since the files are
> opened and closed more often) ...
>

@drink: I didn't know about that but searching a bit it seems that unfortunately there's a license problem.
You can read more here [1] and here [2].

[1]: https://bugs.launchpad.net/ubuntu/+source/bittorrent/+bug/58917
[2]: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=298814

In that case, I give up. It might be better to eliminate it than keep
the antique edition, which only confuses the issue IMO.

Thank you for posting this bug.

Is this still an issue in Lucid?

unassigning. we will demote bittorrent to universe for natty. largely replaced by bittornado

Given comments 18 and 19, and no activity in several years, I think it's clear this is not something that will be fixed here at the distro level, so I'm going to go ahead and pull the plug on it. It sounds like the issue is a legitimate one, so am setting to wontfix rather than invalid.