Duplicity

Merge lp:~brandon753-ba/duplicity/aws-glacier into lp:~duplicity-team/duplicity/0.8-series

  1. aws-glacier
  2. Merge into 0.8-series
Proposed by edso
Status: Merged
Merged at revision: 1357
Proposed branch: lp:~brandon753-ba/duplicity/aws-glacier
Merge into: lp:~duplicity-team/duplicity/0.8-series
Diff against target: 64 lines (+22/-0)
4 files modified
bin/duplicity.1 (+14/-0)
duplicity/backends/_boto_single.py (+2/-0)
duplicity/commandline.py (+3/-0)
duplicity/globals.py (+3/-0)
To merge this branch: bzr merge lp:~brandon753-ba/duplicity/aws-glacier
Reviewer Review Type Date Requested Status
edso Approve
Brandon Anderson (community) Needs Resubmitting
Review via email: mp+364306@code.launchpad.net

Commit message

Adds support for for a command line option to store data on AWS S3 Glacier.

To post a comment you must log in.
Revision history for this message
edso (ed.so) wrote :

hey Brandon,

opened a merge request so the branch becomes commentable. looks good to me.

what it's lacking is the documentation of the parameter in the manpage 'bin/duplicity.1' though.

..ede/duply.net

review: Needs Fixing
lp:~brandon753-ba/duplicity/aws-glacier updated
1354. By Brandon Anderson

Added documentation on how to use the new AWS S3 Glacier option.

Revision history for this message
Brandon Anderson (brandon753-ba) wrote :

Hey Edso,

I just pushed a commit adding documentation to the new feature. Let me know if you need anything else or if I need to do anything different on my end as I have never used bzr or launchpad before only git and svn. Also I am not super happy with this overall patch because while its small and noninvasive ideally there should be a way to use a local copy of the manifest file and not have to store the remote one on S3 Standard instead of glacier. If you have any advice/ideas how that could be done without a lot of changes I would love to hear it. Also while looking at this code it does not seem there is any command checking for exclusive storage options. If I where to enter in duplicity with the options '--s3-use-ia --s3-use-rrs --s3-use-onezone-ia' it looks like it would happily take all the options without throwing an error and store everything on Reduced Redundancy since that is the first option checked in the if statement. I have not looked super deep into this so maybe its checked for elsewhere but it stood out when I made the changes I did.

Sincerely,

Brandon

review: Needs Resubmitting
Revision history for this message
edso (ed.so) wrote :

> Hey Edso,
>
> I just pushed a commit adding documentation to the new feature. Let me know if you need anything else or if I need to do anything different on my end as I have never used bzr or launchpad before only git and svn.

so far no.

>Also I am not super happy with this overall patch because while its small and noninvasive ideally there should be a way to use a local copy of the manifest file and not have to store the remote one on S3 Standard instead of glacier. If you have any advice/ideas how that could be done without a lot of changes I would love to hear it.

restoring from remote usually assumes you have no local copy. so why would you expect a local manifest to be usable with remote volumes?

>Also while looking at this code it does not seem there is any command checking for exclusive storage options. If I where to enter in duplicity with the options '--s3-use-ia --s3-use-rrs --s3-use-onezone-ia' it looks like it would happily take all the options without throwing an error and store everything on Reduced Redundancy since that is the first option checked in the if statement. I have not looked super deep into this so maybe its checked for elsewhere but it stood out when I made the changes I did.

yeah. saw that as well. generally, as these options are excluding each other they should go into one switch. preferably like --s3-storage-class=glacier using the string that is used by the API as well. if you'd like o tackle that youre welcome. keep backward compatibility though. meaning the other parameters setting the new var.

thx.. ede/duply.net

review: Approve
Revision history for this message
Brandon Anderson (brandon753-ba) wrote :

> restoring from remote usually assumes you have no local copy. so why would you
> expect a local manifest to be usable with remote volumes?

I agree that restoring from remote would assume the local manifest is unavailable requiring access to the remote copy, but I was referring to the ability to do incremental backups without having to pull the remote manifest from glacier or keep it stored in S3. If you are doing an incremental backup from a source wouldn't a reasonable assumption be that a local manifest file could be used? Currently duplicity tries to pull the remote manifest in order to do a incremental backup and because of how AWS glacier works it hangs, which is why I have the conditional store the manifest in standard S3 storage, but ideally all files would be stored in glacier and a local manifest could be used for incremental backups.

> >Also while looking at this code it does not seem there is any command
> checking for exclusive storage options. If I where to enter in duplicity with
> the options '--s3-use-ia --s3-use-rrs --s3-use-onezone-ia' it looks like it
> would happily take all the options without throwing an error and store
> everything on Reduced Redundancy since that is the first option checked in the
> if statement. I have not looked super deep into this so maybe its checked for
> elsewhere but it stood out when I made the changes I did.
>
> yeah. saw that as well. generally, as these options are excluding each other
> they should go into one switch. preferably like --s3-storage-class=glacier
> using the string that is used by the API as well. if you'd like o tackle that
> youre welcome. keep backward compatibility though. meaning the other
> parameters setting the new var.

Yea if I get some free time I will push a patch for that.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'bin/duplicity.1'
2--- bin/duplicity.1 2019-01-26 16:36:27 +0000
3+++ bin/duplicity.1 2019-03-13 21:20:20 +0000
4@@ -782,6 +782,20 @@
5 data in one Availability Zone.
6
7 .TP
8+.BI "--s3-use-glacier"
9+Store volumes using Glacier S3 when uploading to Amazon S3. This storage class
10+has a lower cost of storage but a higher per-request cost along with delays
11+of up to 12 hours from the time of retrieval request. This storage cost is
12+calculated against a 90-day storage minimum. According to Amazon this storage is
13+ideal for data archiving and long-term backup offering 99.999999999% durability.
14+To restore a backup you will have to manually migrate all data stored on AWS
15+Glacier back to Standard S3 and wait for AWS to complete the migration.
16+.B Notice:
17+Duplicity will store the manifest.gpg files from full and incremental backups on
18+AWS S3 standard storage to allow quick retrieval for later incremental backups,
19+all other data is stored in S3 Glacier.
20+
21+.TP
22 .BI "--s3-use-multiprocessing"
23 Allow multipart volumne uploads to S3 through multiprocessing. This option
24 requires Python 2.6 and can be used to make uploads to S3 more efficient.
25
26=== modified file 'duplicity/backends/_boto_single.py'
27--- duplicity/backends/_boto_single.py 2019-01-25 17:08:40 +0000
28+++ duplicity/backends/_boto_single.py 2019-03-13 21:20:20 +0000
29@@ -226,6 +226,8 @@
30 storage_class = u'STANDARD_IA'
31 elif globals.s3_use_onezone_ia:
32 storage_class = u'ONEZONE_IA'
33+ elif globals.s3_use_glacier and "manifest.gpg" not in remote_filename:
34+ storage_class = u'GLACIER'
35 else:
36 storage_class = u'STANDARD'
37 log.Info(u"Uploading %s/%s to %s Storage" % (self.straight_url, remote_filename, storage_class))
38
39=== modified file 'duplicity/commandline.py'
40--- duplicity/commandline.py 2019-01-25 17:08:40 +0000
41+++ duplicity/commandline.py 2019-03-13 21:20:20 +0000
42@@ -540,6 +540,9 @@
43 # Whether to use S3 Infrequent Access Storage
44 parser.add_option(u"--s3-use-ia", action=u"store_true")
45
46+ # Whether to use S3 Glacier Storage
47+ parser.add_option(u"--s3-use-glacier", action=u"store_true")
48+
49 # Whether to use S3 One Zone Infrequent Access Storage
50 parser.add_option(u"--s3-use-onezone-ia", action=u"store_true")
51
52
53=== modified file 'duplicity/globals.py'
54--- duplicity/globals.py 2019-01-25 17:08:40 +0000
55+++ duplicity/globals.py 2019-03-13 21:20:20 +0000
56@@ -197,6 +197,9 @@
57 # Whether to use S3 Infrequent Access Storage
58 s3_use_ia = False
59
60+# Whether to use S3 Glacier Storage
61+s3_use_glacier = False
62+
63 # Whether to use S3 One Zone Infrequent Access Storage
64 s3_use_onezone_ia = False
65

Subscribers

People subscribed via source and target branches