Restore from S3 fails with --with-prefix-archive if prefix includes '/'

Bug #1612472 reported by David Cuthbert
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Duplicity
Fix Released
Medium
Unassigned

Bug Description

I'm backing up while writing archive files to a 'glacier/' prefix in S3 to take advantage of S3 lifecycle rules. Writing the backup succeeds, but restores fail because duplicity can't see the archive file in the S3 listing.

Duplicity versions tested: 0.7.08 (from EPEL), 0.7.09 (compiled)
Python version: 2.7.5
OS: RedHat Enterprise Linux 7.2, running in EC2,
Filesystem: Linux xfs

-----
# duplicity --verbosity 9 --file-prefix-archive glacier/ --name lbl-sample-var --no-encryption s3://s3-us-west-2.amazonaws.com/hena-duply-test /tmp/var
Using archive dir: /root/.cache/duplicity/lbl-sample-var
Using backup name: lbl-sample-var
Import of duplicity.backends.acdclibackend Succeeded
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.b2backend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.mediafirebackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Main action: restore
================================================================================
duplicity 0.7.08 (July 02, 2016)
Args: /bin/duplicity --verbosity 9 --file-prefix-archive glacier/ --name lbl-sample-var --no-encryption s3://s3-us-west-2.amazonaws.com/hena-duply-test /tmp/var
Linux ip-172-31-10-5.us-west-2.compute.internal 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64
/bin/python2 2.7.5 (default, Oct 11 2015, 17:47:16)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)]
================================================================================
Using temporary directory /tmp/duplicity-LNctjz-tempdir
Registering (mkstemp) temporary file /tmp/duplicity-LNctjz-tempdir/mkstemp-9cfCtI-1
Temp has 62803791872 available, backup will use approx 272629760.
Listed s3://s3-us-west-2.amazonaws.com/hena-duply-test/duplicity-full-signatures.20160812T002443Z.sigtar.gz
Listed s3://s3-us-west-2.amazonaws.com/hena-duply-test/duplicity-full.20160812T002443Z.manifest
Local and Remote metadata are synchronized, no sync needed.
Listed s3://s3-us-west-2.amazonaws.com/hena-duply-test/duplicity-full-signatures.20160812T002443Z.sigtar.gz
Listed s3://s3-us-west-2.amazonaws.com/hena-duply-test/duplicity-full.20160812T002443Z.manifest
2 files exist on backend
4 files exist in cache
Extracting backup chains from list of files: [u'duplicity-full-signatures.20160812T002443Z.sigtar.gz', u'duplicity-full.20160812T002443Z.manifest']
File duplicity-full-signatures.20160812T002443Z.sigtar.gz is not part of a known set; creating new set
Creating BackupSet with backend <duplicity.backends._boto_single.BotoBackend object at 0x1013b90>
Ignoring file (rejected by backup set) 'duplicity-full-signatures.20160812T002443Z.sigtar.gz'
File duplicity-full.20160812T002443Z.manifest is not part of a known set; creating new set
Creating BackupSet with backend <duplicity.backends._boto_single.BotoBackend object at 0x1013b90>
Processing local manifest /root/.cache/duplicity/lbl-sample-var/duplicity-full.20160812T002443Z.manifest (393724)
Found manifest volume 1
Found 1 volumes in manifest
Found backup chain [Thu Aug 11 20:24:43 2016]-[Thu Aug 11 20:24:43 2016]
Last full backup date: Thu Aug 11 20:24:43 2016
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /root/.cache/duplicity/lbl-sample-var

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Thu Aug 11 20:24:43 2016
Chain end time: Thu Aug 11 20:24:43 2016
Number of contained backup sets: 1
Total number of contained volumes: 0
 Type of backup set: Time: Num volumes:
-------------------------
No orphaned or incomplete backup sets found.
Processing local manifest /root/.cache/duplicity/lbl-sample-var/duplicity-full.20160812T002443Z.manifest (393724)
Found manifest volume 1
Found 1 volumes in manifest
Releasing lockfile /root/.cache/duplicity/lbl-sample-var/lockfile.lock
Removing still remembered temporary file /tmp/duplicity-LNctjz-tempdir/mkstemp-9cfCtI-1
Traceback (most recent call last):
  File "/bin/duplicity", line 1547, in <module>
    with_tempdir(main)
  File "/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/bin/duplicity", line 1392, in main
    do_backup(action)
  File "/bin/duplicity", line 1469, in do_backup
    restore(col_stats)
  File "/bin/duplicity", line 731, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib64/python2.7/site-packages/duplicity/patchdir.py", line 560, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib64/python2.7/site-packages/duplicity/patchdir.py", line 523, in integrate_patch_iters
    for patch_seq in collated:
  File "/usr/lib64/python2.7/site-packages/duplicity/patchdir.py", line 389, in yield_tuples
    setrorps(overflow, elems)
  File "/usr/lib64/python2.7/site-packages/duplicity/patchdir.py", line 378, in setrorps
    elems[i] = iter_list[i].next()
  File "/usr/lib64/python2.7/site-packages/duplicity/patchdir.py", line 121, in difftar2path_iter
    tarinfo_list = [tar_iter.next()]
  File "/usr/lib64/python2.7/site-packages/duplicity/patchdir.py", line 339, in next
    self.set_tarfile()
  File "/usr/lib64/python2.7/site-packages/duplicity/patchdir.py", line 333, in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "/bin/duplicity", line 768, in get_fileobj_iter
    backup_set.volume_name_dict[vol_num],
KeyError: 1
-----

The problem is in the _boto_single backend, which restricts the bucket listing to files not containing a '/'. The attached patch fixes it (but might introduce other performance issues that you're aware of that I'm not):

Revision history for this message
David Cuthbert (dacut) wrote :
Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote : Re: [Bug 1612472] [NEW] Restore from S3 fails with --with-prefix-archive if prefix includes '/'
Download full text (15.0 KiB)

Patch applied by hand. It was reversed and had tabs.

On Thu, Aug 11, 2016 at 7:42 PM, David Cuthbert <email address hidden> wrote:

> Public bug reported:
>
> I'm backing up while writing archive files to a 'glacier/' prefix in S3
> to take advantage of S3 lifecycle rules. Writing the backup succeeds,
> but restores fail because duplicity can't see the archive file in the S3
> listing.
>
> Duplicity versions tested: 0.7.08 (from EPEL), 0.7.09 (compiled)
> Python version: 2.7.5
> OS: RedHat Enterprise Linux 7.2, running in EC2,
> Filesystem: Linux xfs
>
> -----
> # duplicity --verbosity 9 --file-prefix-archive glacier/ --name
> lbl-sample-var --no-encryption s3://s3-us-west-2.amazonaws.
> com/hena-duply-test /tmp/var
> Using archive dir: /root/.cache/duplicity/lbl-sample-var
> Using backup name: lbl-sample-var
> Import of duplicity.backends.acdclibackend Succeeded
> Import of duplicity.backends.azurebackend Succeeded
> Import of duplicity.backends.b2backend Succeeded
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.copycombackend Succeeded
> Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.giobackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.hubicbackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.lftpbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.mediafirebackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.multibackend Succeeded
> Import of duplicity.backends.ncftpbackend Succeeded
> Import of duplicity.backends.onedrivebackend Succeeded
> Import of duplicity.backends.par2backend Succeeded
> Import of duplicity.backends.pydrivebackend Succeeded
> Import of duplicity.backends.rsyncbackend Succeeded
> Import of duplicity.backends.ssh_paramiko_backend Succeeded
> Import of duplicity.backends.ssh_pexpect_backend Succeeded
> Import of duplicity.backends.swiftbackend Succeeded
> Import of duplicity.backends.sxbackend Succeeded
> Import of duplicity.backends.tahoebackend Succeeded
> Import of duplicity.backends.webdavbackend Succeeded
> Main action: restore
> ============================================================
> ====================
> duplicity 0.7.08 (July 02, 2016)
> Args: /bin/duplicity --verbosity 9 --file-prefix-archive glacier/ --name
> lbl-sample-var --no-encryption s3://s3-us-west-2.amazonaws.
> com/hena-duply-test /tmp/var
> Linux ip-172-31-10-5.us-west-2.compute.internal 3.10.0-327.el7.x86_64 #1
> SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64
> /bin/python2 2.7.5 (default, Oct 11 2015, 17:47:16)
> [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)]
> ============================================================
> ====================
> Using temporary directory /tmp/duplicity-LNctjz-tempdir
> Registering (mkstemp) temporary file /tmp/duplicity-LNctjz-tempdir/
> mkstemp-9cfCtI-1
> Temp has 62803791872 available, backup will use approx 272629760.
> List...

Revision history for this message
David Cuthbert (dacut) wrote :

Sorry about that. I thought I had checked for that. (I had to author/test it using a machine that wasn't exactly set up with my normal dev environment.)

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote : Re: [Bug 1612472] Re: Restore from S3 fails with --with-prefix-archive if prefix includes '/'
Download full text (7.9 KiB)

Could have come from Launchpad. No problem.

Thanks for the fix.

On Fri, Aug 12, 2016 at 3:45 PM, David Cuthbert <email address hidden> wrote:

> Sorry about that. I thought I had checked for that. (I had to
> author/test it using a machine that wasn't exactly set up with my normal
> dev environment.)
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1612472
>
> Title:
> Restore from S3 fails with --with-prefix-archive if prefix includes
> '/'
>
> Status in Duplicity:
> Fix Committed
>
> Bug description:
> I'm backing up while writing archive files to a 'glacier/' prefix in
> S3 to take advantage of S3 lifecycle rules. Writing the backup
> succeeds, but restores fail because duplicity can't see the archive
> file in the S3 listing.
>
> Duplicity versions tested: 0.7.08 (from EPEL), 0.7.09 (compiled)
> Python version: 2.7.5
> OS: RedHat Enterprise Linux 7.2, running in EC2,
> Filesystem: Linux xfs
>
> -----
> # duplicity --verbosity 9 --file-prefix-archive glacier/ --name
> lbl-sample-var --no-encryption s3://s3-us-west-2.amazonaws.
> com/hena-duply-test /tmp/var
> Using archive dir: /root/.cache/duplicity/lbl-sample-var
> Using backup name: lbl-sample-var
> Import of duplicity.backends.acdclibackend Succeeded
> Import of duplicity.backends.azurebackend Succeeded
> Import of duplicity.backends.b2backend Succeeded
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.copycombackend Succeeded
> Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.giobackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.hubicbackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.lftpbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.mediafirebackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.multibackend Succeeded
> Import of duplicity.backends.ncftpbackend Succeeded
> Import of duplicity.backends.onedrivebackend Succeeded
> Import of duplicity.backends.par2backend Succeeded
> Import of duplicity.backends.pydrivebackend Succeeded
> Import of duplicity.backends.rsyncbackend Succeeded
> Import of duplicity.backends.ssh_paramiko_backend Succeeded
> Import of duplicity.backends.ssh_pexpect_backend Succeeded
> Import of duplicity.backends.swiftbackend Succeeded
> Import of duplicity.backends.sxbackend Succeeded
> Import of duplicity.backends.tahoebackend Succeeded
> Import of duplicity.backends.webdavbackend Succeeded
> Main action: restore
> ============================================================
> ====================
> duplicity 0.7.08 (July 02, 2016)
> Args: /bin/duplicity --verbosity 9 --file-prefix-archive glacier/ --name
> lbl-sample-var --no-encryption s3://s3-us-west-2.amazonaws.
> com/hena-duply-test /tmp/var
> Linu...

Read more...

Changed in duplicity:
importance: Undecided → Medium
milestone: none → 0.8.00
status: New → Fix Committed
Changed in duplicity:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.