Duplicity

duplicity does not create PAR2 archives when '--par2-options' is used, and does not warn about failure

Bug #1642098 reported by gwern
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Duplicity
Fix Released
Medium
Unassigned

Bug Description

Duplicity appears to fail to create any PAR2 archives if the user specifies an option like increasing RAM use to speed PAR2 up; it also appears to fail to warn about it, and exits normally without reporting the non-creation of PAR2 archives.

---

I installed a new version of duplicity today to get PAR2 & Backblaze support, the `ubuntu2` .deb:

    $ uname -a
    Linux craft 4.2.0-42-generic #49-Ubuntu SMP Tue Jun 28 21:26:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
    $ duplicity --version
    duplicity 0.7.06

Successful run with default PAR2 options:

    $ rm /media/gwern/My\ Book/duplicity2/*; duplicity full --use-agent --sign-key=CBE7F55C --encrypt-key=CBE7F55C --verbosity=9 --volsize=1000 /home/gwern/dnm/ par2+file:///media/gwern/My\ Book/duplicity2/ ; par2verify /media/gwern/My\ Book/duplicity2/*.par2; ls /media/gwern/My\ Book/duplicity2/ ; alert
    Using archive dir: /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787
    Using backup name: 9c8ed3c601f6078e0c518787b9a8f787
    Import of duplicity.backends.azurebackend Succeeded
    Import of duplicity.backends.b2backend Succeeded
    Import of duplicity.backends.botobackend Succeeded
    Import of duplicity.backends.cfbackend Succeeded
    Import of duplicity.backends.copycombackend Succeeded
    Import of duplicity.backends.dpbxbackend Succeeded
    Import of duplicity.backends.gdocsbackend Succeeded
    Import of duplicity.backends.giobackend Succeeded
    Import of duplicity.backends.hsibackend Succeeded
    Import of duplicity.backends.hubicbackend Succeeded
    Import of duplicity.backends.imapbackend Succeeded
    Import of duplicity.backends.lftpbackend Succeeded
    Import of duplicity.backends.localbackend Succeeded
    Import of duplicity.backends.megabackend Succeeded
    Import of duplicity.backends.multibackend Succeeded
    Import of duplicity.backends.ncftpbackend Succeeded
    Import of duplicity.backends.onedrivebackend Succeeded
    Import of duplicity.backends.par2backend Succeeded
    Import of duplicity.backends.pydrivebackend Succeeded
    Import of duplicity.backends.rsyncbackend Succeeded
    Import of duplicity.backends.ssh_paramiko_backend Succeeded
    Import of duplicity.backends.ssh_pexpect_backend Succeeded
    Import of duplicity.backends.swiftbackend Succeeded
    Import of duplicity.backends.sxbackend Succeeded
    Import of duplicity.backends.tahoebackend Succeeded
    Import of duplicity.backends.webdavbackend Succeeded
    Main action: full
    ================================================================================
    duplicity 0.7.06 (December 07, 2015)
    Args: /usr/bin/duplicity full --use-agent --sign-key=CBE7F55C --encrypt-key=CBE7F55C --verbosity=9 --volsize=1000 /home/gwern/dnm/ par2+file:///media/gwern/My Book/duplicity2/
    Linux craft 4.2.0-42-generic #49-Ubuntu SMP Tue Jun 28 21:26:26 UTC 2016 x86_64 x86_64
    /usr/bin/python2 2.7.10 (default, Oct 14 2015, 16:09:02)
    [GCC 5.2.1 20151010]
    ================================================================================
    Using temporary directory /tmp/duplicity-WrL5Rr-tempdir
    Registering (mkstemp) temporary file /tmp/duplicity-WrL5Rr-tempdir/mkstemp-v2toCa-1
    Temp has 182325923840 available, backup will use approx 1363148800.
    Synchronizing remote metadata to local cache...
    Deleting local /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-full-signatures.20161116T005517Z.sigtar.gpg (not authoritative at backend).
    Deleting local /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-full-signatures.20161116T005536Z.sigtar.gz (not authoritative at backend).
    Deleting local /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-full.20161116T005517Z.manifest (not authoritative at backend).
    Deleting local /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-full.20161116T005536Z.manifest.gpg (not authoritative at backend).
    0 files exist on backend
    4 files exist in cache
    Extracting backup chains from list of files: []
    Warning, found signatures but no corresponding backup files
    Last full backup date: none
    Collection Status
    -----------------
    Connecting with backend: BackendWrapper
    Archive dir: /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787

    Found 0 secondary backup chains.
    No backup chains with active signatures found
    No orphaned or incomplete backup sets found.
    Using temporary directory /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-DmUq5D-tempdir
    Registering (mktemp) temporary file /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-DmUq5D-tempdir/mktemp-br1Phg-1
    Using temporary directory /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-5o1Ley-tempdir
    Registering (mktemp) temporary file /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-5o1Ley-tempdir/mktemp-hL0cW6-1
    AsyncScheduler: instantiating at concurrency 0
    Registering (mktemp) temporary file /tmp/duplicity-WrL5Rr-tempdir/mktemp-5cQ1oS-2
    Selecting /home/gwern/dnm
    Comparing . and None
    Getting delta of (. dir) and None
    A .
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU
    Comparing 3-16-cr-00109-SRU and None
    Getting delta of (3-16-cr-00109-SRU dir) and None
    A 3-16-cr-00109-SRU
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-1.pdf
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-1.pdf
    Comparing 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-1.pdf and None
    Getting delta of (3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-1.pdf reg) and None
    A 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-1.pdf
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-2.pdf
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-2.pdf
    Comparing 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-2.pdf and None
    Getting delta of (3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-2.pdf reg) and None
    A 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-2.pdf
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-3.pdf
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-3.pdf
    Comparing 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-3.pdf and None
    Getting delta of (3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-3.pdf reg) and None
    A 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-3.pdf
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-main.pdf
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-main.pdf
    Comparing 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-main.pdf and None
    Getting delta of (3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-main.pdf reg) and None
    A 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-main.pdf
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU.zip
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU.zip
    Comparing 3-16-cr-00109-SRU.zip and None
    Getting delta of (3-16-cr-00109-SRU.zip reg) and None
    A 3-16-cr-00109-SRU.zip
    Removing still remembered temporary file /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-DmUq5D-tempdir/mktemp-br1Phg-1
    Removing still remembered temporary file /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-5o1Ley-tempdir/mktemp-hL0cW6-1
    AsyncScheduler: running task synchronously (asynchronicity disabled)
    Writing duplicity-full.20161116T005642Z.vol1.difftar.gpg
    Making directory /tmp/duplicity-WrL5Rr-tempdir/duplicity_temp.1
    Create Par2 recovery files
    Deleting /tmp/duplicity-WrL5Rr-tempdir/duplicity_temp.1/duplicity-full.20161116T005642Z.vol1.difftar.gpg
    Deleting tree /tmp/duplicity-WrL5Rr-tempdir/duplicity_temp.1
    Selecting /tmp/duplicity-WrL5Rr-tempdir/duplicity_temp.1
    Selection: examining path /tmp/duplicity-WrL5Rr-tempdir/duplicity_temp.1/duplicity-full.20161116T005642Z.vol1.difftar.gpg.par2
    Selection: + no selection functions found. Including
    Selecting /tmp/duplicity-WrL5Rr-tempdir/duplicity_temp.1/duplicity-full.20161116T005642Z.vol1.difftar.gpg.par2
    Deleting /tmp/duplicity-WrL5Rr-tempdir/duplicity_temp.1/duplicity-full.20161116T005642Z.vol1.difftar.gpg.par2
    Selection: examining path /tmp/duplicity-WrL5Rr-tempdir/duplicity_temp.1/duplicity-full.20161116T005642Z.vol1.difftar.gpg.vol000+200.par2
    Selection: + no selection functions found. Including
    Selecting /tmp/duplicity-WrL5Rr-tempdir/duplicity_temp.1/duplicity-full.20161116T005642Z.vol1.difftar.gpg.vol000+200.par2
    Deleting /tmp/duplicity-WrL5Rr-tempdir/duplicity_temp.1/duplicity-full.20161116T005642Z.vol1.difftar.gpg.vol000+200.par2
    Deleting /tmp/duplicity-WrL5Rr-tempdir/duplicity_temp.1
    Deleting /tmp/duplicity-WrL5Rr-tempdir/mktemp-5cQ1oS-2
    Forgetting temporary file /tmp/duplicity-WrL5Rr-tempdir/mktemp-5cQ1oS-2
    AsyncScheduler: task completed successfully
    Processed volume 1
    Making directory /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Create Par2 recovery files
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full-signatures.20161116T005642Z.sigtar.gpg
    Deleting tree /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Selecting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Selection: examining path /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full-signatures.20161116T005642Z.sigtar.gpg.par2
    Selection: + no selection functions found. Including
    Selecting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full-signatures.20161116T005642Z.sigtar.gpg.par2
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full-signatures.20161116T005642Z.sigtar.gpg.par2
    Selection: examining path /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full-signatures.20161116T005642Z.sigtar.gpg.vol000+190.par2
    Selection: + no selection functions found. Including
    Selecting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full-signatures.20161116T005642Z.sigtar.gpg.vol000+190.par2
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full-signatures.20161116T005642Z.sigtar.gpg.vol000+190.par2
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Making directory /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Create Par2 recovery files
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full.20161116T005642Z.manifest.gpg
    Deleting tree /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Selecting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Selection: examining path /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full.20161116T005642Z.manifest.gpg.par2
    Selection: + no selection functions found. Including
    Selecting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full.20161116T005642Z.manifest.gpg.par2
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full.20161116T005642Z.manifest.gpg.par2
    Selection: examining path /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full.20161116T005642Z.manifest.gpg.vol00+34.par2
    Selection: + no selection functions found. Including
    Selecting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full.20161116T005642Z.manifest.gpg.vol00+34.par2
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full.20161116T005642Z.manifest.gpg.vol00+34.par2
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    1 file exists on backend
    8 files exist in cache
    Extracting backup chains from list of files: [u'duplicity-full.20161116T005642Z.vol1.difftar.gpg']
    File duplicity-full.20161116T005642Z.vol1.difftar.gpg is not part of a known set; creating new set
    Warning, found incomplete backup sets, probably left from aborted session
    --------------[ Backup Statistics ]--------------
    StartTime 1479257802.19 (Tue Nov 15 19:56:42 2016)
    EndTime 1479257802.76 (Tue Nov 15 19:56:42 2016)
    ElapsedTime 0.57 (0.57 seconds)
    SourceFiles 7
    SourceFileSize 11078834 (10.6 MB)
    NewFiles 7
    NewFileSize 11078834 (10.6 MB)
    DeletedFiles 0
    ChangedFiles 0
    ChangedFileSize 0 (0 bytes)
    ChangedDeltaSize 0 (0 bytes)
    DeltaEntries 7
    RawDeltaSize 11070642 (10.6 MB)
    TotalDestinationSizeChange 8573459 (8.18 MB)
    Errors 0
    -------------------------------------------------

    Releasing lockfile /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/lockfile.lock
    Removing still remembered temporary file /tmp/duplicity-WrL5Rr-tempdir/mkstemp-v2toCa-1
    Loading "duplicity-full.20161116T005642Z.vol1.difftar.gpg.par2".
    Loaded 4 new packets
    Loading "duplicity-full.20161116T005642Z.vol1.difftar.gpg.vol000+200.par2".
    Loaded 200 new packets including 200 recovery blocks

    There are 1 recoverable files and 0 other files.
    The block size used was 4288 bytes.
    There are a total of 2000 data blocks.
    The total size of the data files is 8573459 bytes.

    Verifying source files:

    Target: "duplicity-full.20161116T005642Z.vol1.difftar.gpg" - found.

    All files are correct, repair is not required.
    duplicity-full.20161116T005642Z.vol1.difftar.gpg
    duplicity-full.20161116T005642Z.vol1.difftar.gpg.par2
    duplicity-full.20161116T005642Z.vol1.difftar.gpg.vol000+200.par2

Should I try tweaking the redundancy, it works fine with `--par2-redundancy 5` etc. So that particular PAR2 Duplicity option is working.

It always fails when I ask Duplicity to pass onto PAR2 an option, the -m option which increases RAM use by PAR2:

       -m<n> Memory (in MB) to use

Due to the space-time tradeoff, PAR2 runs *much* faster (as in, several times faster) if you give it a more reasonable amount of RAM commensurate with modern standards, like 2GB. I always use it because it makes such a difference and I always have a few GB of RAM to spare. (It might not be a bad idea at all to have duplicity call PAR2 with a changed default....) However when I try:

    $ rm /media/gwern/My\ Book/duplicity2/*; duplicity full --use-agent --sign-key=CBE7F55C --encrypt-key=CBE7F55C --verbosity=9 --volsize=1000 --par2-options -m2000 /home/gwern/dnm/ par2+file:///media/gwern/My\ Book/duplicity2/ ; par2verify /media/gwern/My\ Book/duplicity2/*.par2; ls /media/gwern/My\ Book/duplicity2/ ; alert
    Using archive dir: /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787
    Using backup name: 9c8ed3c601f6078e0c518787b9a8f787
    Import of duplicity.backends.azurebackend Succeeded
    Import of duplicity.backends.b2backend Succeeded
    Import of duplicity.backends.botobackend Succeeded
    Import of duplicity.backends.cfbackend Succeeded
    Import of duplicity.backends.copycombackend Succeeded
    Import of duplicity.backends.dpbxbackend Succeeded
    Import of duplicity.backends.gdocsbackend Succeeded
    Import of duplicity.backends.giobackend Succeeded
    Import of duplicity.backends.hsibackend Succeeded
    Import of duplicity.backends.hubicbackend Succeeded
    Import of duplicity.backends.imapbackend Succeeded
    Import of duplicity.backends.lftpbackend Succeeded
    Import of duplicity.backends.localbackend Succeeded
    Import of duplicity.backends.megabackend Succeeded
    Import of duplicity.backends.multibackend Succeeded
    Import of duplicity.backends.ncftpbackend Succeeded
    Import of duplicity.backends.onedrivebackend Succeeded
    Import of duplicity.backends.par2backend Succeeded
    Import of duplicity.backends.pydrivebackend Succeeded
    Import of duplicity.backends.rsyncbackend Succeeded
    Import of duplicity.backends.ssh_paramiko_backend Succeeded
    Import of duplicity.backends.ssh_pexpect_backend Succeeded
    Import of duplicity.backends.swiftbackend Succeeded
    Import of duplicity.backends.sxbackend Succeeded
    Import of duplicity.backends.tahoebackend Succeeded
    Import of duplicity.backends.webdavbackend Succeeded
    Main action: full
    ================================================================================
    duplicity 0.7.06 (December 07, 2015)
    Args: /usr/bin/duplicity full --use-agent --sign-key=CBE7F55C --encrypt-key=CBE7F55C --verbosity=9 --volsize=1000 --par2-options -m2000 /home/gwern/dnm/ par2+file:///media/gwern/My Book/duplicity2/
    Linux craft 4.2.0-42-generic #49-Ubuntu SMP Tue Jun 28 21:26:26 UTC 2016 x86_64 x86_64
    /usr/bin/python2 2.7.10 (default, Oct 14 2015, 16:09:02)
    [GCC 5.2.1 20151010]
    ================================================================================
    Using temporary directory /tmp/duplicity-2Y7f4X-tempdir
    Registering (mkstemp) temporary file /tmp/duplicity-2Y7f4X-tempdir/mkstemp-vrrmsY-1
    Temp has 182325932032 available, backup will use approx 1363148800.
    Synchronizing remote metadata to local cache...
    Deleting local /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-full-signatures.20161116T005319Z.sigtar.gz (not authoritative at backend).
    Deleting local /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-full-signatures.20161116T005333Z.sigtar.gpg (not authoritative at backend).
    Deleting local /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-full.20161116T005319Z.manifest (not authoritative at backend).
    Deleting local /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-full.20161116T005333Z.manifest.gpg (not authoritative at backend).
    0 files exist on backend
    4 files exist in cache
    Extracting backup chains from list of files: []
    Warning, found signatures but no corresponding backup files
    Last full backup date: none
    Collection Status
    -----------------
    Connecting with backend: BackendWrapper
    Archive dir: /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787

    Found 0 secondary backup chains.
    No backup chains with active signatures found
    No orphaned or incomplete backup sets found.
    Using temporary directory /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-IfAgkn-tempdir
    Registering (mktemp) temporary file /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-IfAgkn-tempdir/mktemp-Wrj8tw-1
    Using temporary directory /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-uvnjxK-tempdir
    Registering (mktemp) temporary file /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-uvnjxK-tempdir/mktemp-yXdi3o-1
    AsyncScheduler: instantiating at concurrency 0
    Registering (mktemp) temporary file /tmp/duplicity-2Y7f4X-tempdir/mktemp-B2yYEE-2
    Selecting /home/gwern/dnm
    Comparing . and None
    Getting delta of (. dir) and None
    A .
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU
    Comparing 3-16-cr-00109-SRU and None
    Getting delta of (3-16-cr-00109-SRU dir) and None
    A 3-16-cr-00109-SRU
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-1.pdf
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-1.pdf
    Comparing 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-1.pdf and None
    Getting delta of (3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-1.pdf reg) and None
    A 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-1.pdf
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-2.pdf
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-2.pdf
    Comparing 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-2.pdf and None
    Getting delta of (3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-2.pdf reg) and None
    A 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-2.pdf
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-3.pdf
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-3.pdf
    Comparing 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-3.pdf and None
    Getting delta of (3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-3.pdf reg) and None
    A 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-3.pdf
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-main.pdf
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-main.pdf
    Comparing 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-main.pdf and None
    Getting delta of (3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-main.pdf reg) and None
    A 3-16-cr-00109-SRU/3-16-cr-00109-SRU-1-main.pdf
    Selection: examining path /home/gwern/dnm/3-16-cr-00109-SRU.zip
    Selection: + no selection functions found. Including
    Selecting /home/gwern/dnm/3-16-cr-00109-SRU.zip
    Comparing 3-16-cr-00109-SRU.zip and None
    Getting delta of (3-16-cr-00109-SRU.zip reg) and None
    A 3-16-cr-00109-SRU.zip
    Removing still remembered temporary file /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-IfAgkn-tempdir/mktemp-Wrj8tw-1
    Removing still remembered temporary file /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity-uvnjxK-tempdir/mktemp-yXdi3o-1
    AsyncScheduler: running task synchronously (asynchronicity disabled)
    Writing duplicity-full.20161116T005401Z.vol1.difftar.gpg
    Making directory /tmp/duplicity-2Y7f4X-tempdir/duplicity_temp.1
    Create Par2 recovery files
    Deleting /tmp/duplicity-2Y7f4X-tempdir/duplicity_temp.1/duplicity-full.20161116T005401Z.vol1.difftar.gpg
    Deleting tree /tmp/duplicity-2Y7f4X-tempdir/duplicity_temp.1
    Selecting /tmp/duplicity-2Y7f4X-tempdir/duplicity_temp.1
    Deleting /tmp/duplicity-2Y7f4X-tempdir/duplicity_temp.1
    Deleting /tmp/duplicity-2Y7f4X-tempdir/mktemp-B2yYEE-2
    Forgetting temporary file /tmp/duplicity-2Y7f4X-tempdir/mktemp-B2yYEE-2
    AsyncScheduler: task completed successfully
    Processed volume 1
    Making directory /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Create Par2 recovery files
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full-signatures.20161116T005401Z.sigtar.gpg
    Deleting tree /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Selecting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Making directory /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Create Par2 recovery files
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1/duplicity-full.20161116T005401Z.manifest.gpg
    Deleting tree /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Selecting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    Deleting /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/duplicity_temp.1
    1 file exists on backend
    8 files exist in cache
    Extracting backup chains from list of files: [u'duplicity-full.20161116T005401Z.vol1.difftar.gpg']
    File duplicity-full.20161116T005401Z.vol1.difftar.gpg is not part of a known set; creating new set
    Warning, found incomplete backup sets, probably left from aborted session
    --------------[ Backup Statistics ]--------------
    StartTime 1479257641.59 (Tue Nov 15 19:54:01 2016)
    EndTime 1479257642.18 (Tue Nov 15 19:54:02 2016)
    ElapsedTime 0.59 (0.59 seconds)
    SourceFiles 7
    SourceFileSize 11078834 (10.6 MB)
    NewFiles 7
    NewFileSize 11078834 (10.6 MB)
    DeletedFiles 0
    ChangedFiles 0
    ChangedFileSize 0 (0 bytes)
    ChangedDeltaSize 0 (0 bytes)
    DeltaEntries 7
    RawDeltaSize 11070642 (10.6 MB)
    TotalDestinationSizeChange 8573460 (8.18 MB)
    Errors 0
    -------------------------------------------------

    Releasing lockfile /home/gwern/.cache/duplicity/9c8ed3c601f6078e0c518787b9a8f787/lockfile.lock
    Removing still remembered temporary file /tmp/duplicity-2Y7f4X-tempdir/mkstemp-vrrmsY-1
    par2 file must not have a wildcard in it.
    duplicity-full.20161116T005401Z.vol1.difftar.gpg

All variants I tried failed:

--par2-options "m2000"
--par2-options "-m2000"
--par2-options=-m2000
--par2-options=m2000
--par2-options m2000
--par2-options -m2000

Further, duplicity appears to be oblivious to the PAR2 errors or to not be running par2 at all (even debug-level output is unclear on what it's actually doing, as it merely says "Create Par2 recovery files" and continues normally).

Revision history for this message
gwern (gwern0) wrote :

An additional detail: some of the invocations, while failing to produce PAR2s in the backup destination, did dump some .par2 files in ~/, like m2000-q.par2 m2000-q.vol000+190.par2 m2000-q.vol00+34.par2 m4000-q.par2 m4000-q.vol000+163.par2 m4000-q.vol000+200.par2. (Not sure which, I didn't notice them until just now.)

Revision history for this message
gwern (gwern0) wrote :

I've updated my Duplicity installation using the nightly PPA to use 0.7.10+bzr1244; the same behavior occurs (no .par2s in the backup, some .par2s in ~/, no error messages).

Kenneth Loafman (kenneth-loafman)
Changed in duplicity:
status: New → Fix Committed
Kenneth Loafman (kenneth-loafman)
Changed in duplicity:
importance: Undecided → Medium
Revision history for this message
Jeremy Saklad (saklad5) wrote :

The causes of this bug are the hardcoded arguments in the par2 backend. Most notably, “-n1” conflicts with any user-specified arguments for “-l” or “-n”, and makes it impossible to use split par2 archives. Since par2 archives are supposed to be used as split archives, minimizing the amount of recovery blocks that have to be downloaded to repair a problem, this is a major issue.

The arguments are appended onto the hardcoded ones, checking to see if any of them conflict with the defaults. I was able to fix all reported problems by deleting the hardcoded arguments and supplying my own replacements, as a proof-of-concept.

To actually fix this, the --par2-options parameters should be made to override the currently hardcoded defaults.

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote : Re: [Bug 1642098] Re: duplicity does not create PAR2 archives when '--par2-options' is used, and does not warn about failure
Download full text (28.6 KiB)

Have you tried the version in the trunk? It should be fixed.

On Mon, Aug 21, 2017 at 4:12 PM, Jeremy Saklad <email address hidden>
wrote:

> The causes of this bug are the hardcoded arguments in the par2 backend.
> Most notably, “-n1” conflicts with any user-specified arguments for “-l”
> or “-n”, and makes it impossible to use split par2 archives. Since par2
> archives are supposed to be used as split archives, minimizing the
> amount of recovery blocks that have to be downloaded to repair a
> problem, this is a major issue.
>
> The arguments are appended onto the hardcoded ones, checking to see if
> any of them conflict with the defaults. I was able to fix all reported
> problems by deleting the hardcoded arguments and supplying my own
> replacements, as a proof-of-concept.
>
> To actually fix this, the --par2-options parameters should be made to
> override the currently hardcoded defaults.
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1642098
>
> Title:
> duplicity does not create PAR2 archives when '--par2-options' is used,
> and does not warn about failure
>
> Status in Duplicity:
> Fix Committed
>
> Bug description:
> Duplicity appears to fail to create any PAR2 archives if the user
> specifies an option like increasing RAM use to speed PAR2 up; it also
> appears to fail to warn about it, and exits normally without reporting
> the non-creation of PAR2 archives.
>
> ---
>
> I installed a new version of duplicity today to get PAR2 & Backblaze
> support, the `ubuntu2` .deb:
>
> $ uname -a
> Linux craft 4.2.0-42-generic #49-Ubuntu SMP Tue Jun 28 21:26:26 UTC
> 2016 x86_64 x86_64 x86_64 GNU/Linux
> $ duplicity --version
> duplicity 0.7.06
>
> Successful run with default PAR2 options:
>
> $ rm /media/gwern/My\ Book/duplicity2/*; duplicity full --use-agent
> --sign-key=CBE7F55C --encrypt-key=CBE7F55C --verbosity=9 --volsize=1000
> /home/gwern/dnm/ par2+file:///media/gwern/My\ Book/duplicity2/ ; par2verify
> /media/gwern/My\ Book/duplicity2/*.par2; ls /media/gwern/My\
> Book/duplicity2/ ; alert
> Using archive dir: /home/gwern/.cache/duplicity/
> 9c8ed3c601f6078e0c518787b9a8f787
> Using backup name: 9c8ed3c601f6078e0c518787b9a8f787
> Import of duplicity.backends.azurebackend Succeeded
> Import of duplicity.backends.b2backend Succeeded
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.copycombackend Succeeded
> Import of duplicity.backends.dpbxbackend Succeeded
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.giobackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.hubicbackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.lftpbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.multibackend Succeed...

Revision history for this message
gwern (gwern0) wrote :

I've updated my installation to duplicity 0.7.13.1+bzr1308 and retested it. The PAR2 files are now generated when I try a test command like this (backing up 22GB of my music):

    duplicity full --verbosity=9 --volsize=5000 --par2-options -m15000 /home/gwern/music/ par2+file:///media/gwern/My\ Book/duplicity2/

The RAM option shows up in the invocation when I watch in htop, memory consumption increases with large volsizes, and par2verify is successful but only if I use a for loop.

---

Looking at the generated par2 files and noting the interleaving of PAR2 with the backup volumes, I am a little confused. It seems that duplicity is not generating a PAR2 for the set of generated volumes, but is generating a single PAR2 for each volume individually? Doesn't this largely defeat the entire point of generating PAR2? If a single volume is lost for any reason, it cannot be repaired using a 10% PAR2 of it on its own, thereby corrupting the entire backup and all incrementals after it; while it could have been repaired if PAR2 was being used normally on the entire set of generated volumes. The only thing that you could repair with per-volume PAR2s would be, I dunno, some bitflips or (very small) amounts of truncation. Am I invoking PAR2 wrong in the duplicity options?

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

It was never intended to make a backup-wide par2 file, rather to make it easier to recover from the bit errors introduced by networking. You can use the par2 options to increase the redundancy percentage or redundancy target size, if you want. As far as I know we've only seen corruption in single volumes, not across multiple volumes.

Revision history for this message
gwern (gwern0) wrote :

> As far as I know we've only seen corruption in single volumes, not across multiple volumes.

If you are using incremental backups, doesn't a loss of any volume entail a loss of the multiple volumes afterwards which depend on it...?

In any case, I can safely say that loss of individual volumes, rather than bit errors in networking (how would that even happen given TCP checksums etc?), is the only kind of corruption I've ever had with duplicity, as that is typically how failure happens on my DVDs/BD/HDDs - the medium goes bad and now you cannot read a file, as opposed to reading the file with 1 bitflip. This would be irrecoverable with anything less than 100% redundancy. I just burned a set of BDs, and a number of the split files couldn't be read back; which is fine, as I could reburn them onto the next BD and I had generated a very large amount of PAR2 redundancy over the *entire* set of split files, as PAR2 is intended to be used, and lost files can easily be reconstructed.

There should at least be a warning in the man page about the generated PAR2 being useful only for correcting bitflips or very small amounts of corruption and not major data loss, as the current 'a note on par2 wrapper backend' misleadingly implies that PAR2 is being used normally.

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

As to networking, yes even TCP/UDP have problems, especially when an intermediate router along your route has had checksumming turned off to speed up throughput. I discovered that a long time ago when I found I could safely download some things from work and not home, and vice versa. A network engineer filled me in on the practice.

Also, consumer grade memory is not parity checked anymore. The assumption is that errors are infrequent enough that no check is needed. The assumption is correct, for the most part.

As to PAR2 being used across the entire backup, perhaps someone will write an additional option to add that feature. I would appreciate it. The ability to recover an entire missing file would be fantastic.

Kenneth Loafman (kenneth-loafman)
Changed in duplicity:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.