Unable to restore backup due to FD_SETSIZE

What duplicity version? What OS version? What Python version?

sorry for the late reply. Just returned from holiday.

I'm using mageia linux 7, duplicity-0.8.12.1612-1.mga7.x86_64 and python-2.7.17-1.1.mga7.x86_64

As stated above, during incremental restore it looks like each part for the restore is spawned (gpg) and after finishing not closed, so the number of processes increases until "FD_SETSIZE" is reached, which is a fixed limit in c/c++ set to 1024 processes in select. (my assumption)

Try this:
# ulimit -n 65536
# duplicity ...your options...

If that works, set it globally via one of the solutions on the net. Google "linux too many file handles". I am not familiar with your distro, so find the solution closest.

Also, so I can rule out one problem I've encountered from downstream distros, please check the duplicity directory for the file 'gpginterface.py', then check line 696 for threaded_waitpid(). I had to modify the original to harvest the fileid's of completed processes. Some distro's will make the mods back to the original in error, something I've never understood.

If none of the above works, please upgrade to the current version of duplicity. This will assure that any bugs fixed since your release are available and may fix your issue.

There are multiple options both stable and daily:

* Source - https://gitlab.com/duplicity/duplicity
* Stable tarball install - https://launchpad.net/duplicity/+download
* Daily duplicity PPA - https://code.launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-develop-git
* Stable duplicity PPA - https://code.launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-release-git
* Stable snap builds - “sudo snap install duplicity —classic"
* Latest snap builds - “sudo snap install duplicity —classic —edge"
* Latest pip3 builds - “sudo pip3 install duplicity"

Note 1: UNINSTALL duplicity first if it was installed from a different source.. This is due to divergent install locations, especially between repository installs and the other forms.

Note 2: Launchpad PPAs contain builds for Bionic 18.04, Eoan 19.10, and Focal 20.04. Xenial 16.04 works with Snap and Pip installs, but cannot be built under Launchpad PPAs at the moment.

thanks for your reply, unfortunately I can't try it anymore. I've forgotten to deactivate the cleanup script that removed the incremental backups between full's :(
If I have time, I can try to recreate that issue to check if it is solved - but for now the files are really gone :(

Sorry to hear that!

I'm going to leave this open for now. If you have the same issue, please report it.

Ok, this issue hit me again.
setting ulimit does not solve the problem:
I still get:

Letzte vollständige Sicherung: Fri Oct 16 16:10:33 2020
Versuch 1 ist fehlgeschlagen. ValueError: filedescriptor out of range in select()

I've added some debug output from duplicity.
I hope that helps.

At this point I'd suggest an update to duplicity itself. 0.8.12 is old, the new version is 0.8.17. Download instructions are in comment #4 above.

Updated to 0.8.17 - no difference!

duplicity uses python3-3.7.6-1
many sites tell select call is limited to 1024 filedescriptors, even if ulimit is set:

https://stackoverflow.com/questions/7695701/filedescriptor-out-of-range-in-select-when-using-pythons-subprocess-with-rs

https://python-forum.io/Thread-Filedescriptor-out-of-range-in-select

Since I'm no python programmer, I don't where select is used or how to enable "debug" to get a stacktrace when the error occurs.

If you have anything to test, I can check this.

One more thing: Try using the paramiko+sftp backend instead of the
pexpect+sftp backend.

On Thu, Nov 12, 2020 at 10:55 AM M <email address hidden> wrote:

> Updated to 0.8.17 - no difference!
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1895353
>
> Title:
> Unable to restore backup due to FD_SETSIZE
>
> Status in Duplicity:
> Incomplete
>
> Bug description:
> trying to restore a backup shows the following error:
> ValueError: filedescriptor out of range in select()
>
> and then nothing else happens.
>
>
> You can see, ~170 instances of gpg were spawned in order to restore the
> backup. It was not "ideal" having incremental backups for 2 month, but
> still it should be possible to recover backups like this
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1895353/+subscriptions
>

What must be changed to use it?
I get
BackendException: ssh connection to usr@backup:23 failed: Authentication failed.
when I change
--ssh-options="-i /etc/ssh/ssh_host_rsa_key" pexpect+sftp://user@backup:23
to
--ssh-options="-i /etc/ssh/ssh_host_rsa_key" paramiko+sftp://user@backup:23

finally got paramiko working.

at frist it crashed too with "OSError: [Errno 24] Too many open files"

after setting
ulimit -Sn 4096

it works.

Can this be improved not to keep more than 1024 files open at a time?!

This is an ongoing problem due to the chain nature of incremental backups. I've done various things to work around it, but a long backup chain invariably eats up fd's as it grows. My answer is always: Keep the backup chains short, 1 per day for a month, then do a full.