Restore from S3 fails with --with-prefix-archive if prefix includes '/'
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Duplicity |
Fix Released
|
Medium
|
Unassigned |
Bug Description
I'm backing up while writing archive files to a 'glacier/' prefix in S3 to take advantage of S3 lifecycle rules. Writing the backup succeeds, but restores fail because duplicity can't see the archive file in the S3 listing.
Duplicity versions tested: 0.7.08 (from EPEL), 0.7.09 (compiled)
Python version: 2.7.5
OS: RedHat Enterprise Linux 7.2, running in EC2,
Filesystem: Linux xfs
-----
# duplicity --verbosity 9 --file-
Using archive dir: /root/.
Using backup name: lbl-sample-var
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Import of duplicity.
Main action: restore
=======
duplicity 0.7.08 (July 02, 2016)
Args: /bin/duplicity --verbosity 9 --file-
Linux ip-172-
/bin/python2 2.7.5 (default, Oct 11 2015, 17:47:16)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)]
=======
Using temporary directory /tmp/duplicity-
Registering (mkstemp) temporary file /tmp/duplicity-
Temp has 62803791872 available, backup will use approx 272629760.
Listed s3://s3-
Listed s3://s3-
Local and Remote metadata are synchronized, no sync needed.
Listed s3://s3-
Listed s3://s3-
2 files exist on backend
4 files exist in cache
Extracting backup chains from list of files: [u'duplicity-
File duplicity-
Creating BackupSet with backend <duplicity.
Ignoring file (rejected by backup set) 'duplicity-
File duplicity-
Creating BackupSet with backend <duplicity.
Processing local manifest /root/.
Found manifest volume 1
Found 1 volumes in manifest
Found backup chain [Thu Aug 11 20:24:43 2016]-[Thu Aug 11 20:24:43 2016]
Last full backup date: Thu Aug 11 20:24:43 2016
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /root/.
Found 0 secondary backup chains.
Found primary backup chain with matching signature chain:
-------
Chain start time: Thu Aug 11 20:24:43 2016
Chain end time: Thu Aug 11 20:24:43 2016
Number of contained backup sets: 1
Total number of contained volumes: 0
Type of backup set: Time: Num volumes:
-------
No orphaned or incomplete backup sets found.
Processing local manifest /root/.
Found manifest volume 1
Found 1 volumes in manifest
Releasing lockfile /root/.
Removing still remembered temporary file /tmp/duplicity-
Traceback (most recent call last):
File "/bin/duplicity", line 1547, in <module>
with_
File "/bin/duplicity", line 1541, in with_tempdir
fn()
File "/bin/duplicity", line 1392, in main
do_
File "/bin/duplicity", line 1469, in do_backup
restore(
File "/bin/duplicity", line 731, in restore
restore_
File "/usr/lib64/
for ropath in rop_iter:
File "/usr/lib64/
for patch_seq in collated:
File "/usr/lib64/
setrorps(
File "/usr/lib64/
elems[i] = iter_list[i].next()
File "/usr/lib64/
tarinfo_list = [tar_iter.next()]
File "/usr/lib64/
self.
File "/usr/lib64/
self.current_fp = self.fileobj_
File "/bin/duplicity", line 768, in get_fileobj_iter
backup_
KeyError: 1
-----
The problem is in the _boto_single backend, which restricts the bucket listing to files not containing a '/'. The attached patch fixes it (but might introduce other performance issues that you're aware of that I'm not):
Changed in duplicity: | |
status: | Fix Committed → Fix Released |
Patch applied by hand. It was reversed and had tabs.
On Thu, Aug 11, 2016 at 7:42 PM, David Cuthbert <email address hidden> wrote:
> Public bug reported: prefix- archive glacier/ --name us-west- 2.amazonaws. cache/duplicity /lbl-sample- var backends. acdclibackend Succeeded backends. azurebackend Succeeded backends. b2backend Succeeded backends. botobackend Succeeded backends. cfbackend Succeeded backends. copycombackend Succeeded backends. dpbxbackend Failed: No module named dropbox backends. gdocsbackend Succeeded backends. giobackend Succeeded backends. hsibackend Succeeded backends. hubicbackend Succeeded backends. imapbackend Succeeded backends. lftpbackend Succeeded backends. localbackend Succeeded backends. mediafirebacken d Succeeded backends. megabackend Succeeded backends. multibackend Succeeded backends. ncftpbackend Succeeded backends. onedrivebackend Succeeded backends. par2backend Succeeded backends. pydrivebackend Succeeded backends. rsyncbackend Succeeded backends. ssh_paramiko_ backend Succeeded backends. ssh_pexpect_ backend Succeeded backends. swiftbackend Succeeded backends. sxbackend Succeeded backends. tahoebackend Succeeded backends. webdavbackend Succeeded ======= ======= ======= ======= ======= ======= ======= ==== ======= ====== prefix- archive glacier/ --name us-west- 2.amazonaws. 31-10-5. us-west- 2.compute. internal 3.10.0- 327.el7. x86_64 #1 ======= ======= ======= ======= ======= ======= ======= ==== ======= ====== LNctjz- tempdir LNctjz- tempdir/
>
> I'm backing up while writing archive files to a 'glacier/' prefix in S3
> to take advantage of S3 lifecycle rules. Writing the backup succeeds,
> but restores fail because duplicity can't see the archive file in the S3
> listing.
>
> Duplicity versions tested: 0.7.08 (from EPEL), 0.7.09 (compiled)
> Python version: 2.7.5
> OS: RedHat Enterprise Linux 7.2, running in EC2,
> Filesystem: Linux xfs
>
> -----
> # duplicity --verbosity 9 --file-
> lbl-sample-var --no-encryption s3://s3-
> com/hena-duply-test /tmp/var
> Using archive dir: /root/.
> Using backup name: lbl-sample-var
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Import of duplicity.
> Main action: restore
> =======
> =======
> duplicity 0.7.08 (July 02, 2016)
> Args: /bin/duplicity --verbosity 9 --file-
> lbl-sample-var --no-encryption s3://s3-
> com/hena-duply-test /tmp/var
> Linux ip-172-
> SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64
> /bin/python2 2.7.5 (default, Oct 11 2015, 17:47:16)
> [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)]
> =======
> =======
> Using temporary directory /tmp/duplicity-
> Registering (mkstemp) temporary file /tmp/duplicity-
> mkstemp-9cfCtI-1
> Temp has 62803791872 available, backup will use approx 272629760.
> List...