News and announcements
Duplicity 0.6.22 Released
Written for Duplicity by Kenneth Loafman on 2013-08-22
New in v0.6.22 (2013/08/22)
-------
Enhancements:
* Applied patches from Laszlo Ersek to rdiffdir to "consume a chain of sigtar
files in rdiffdir delta mode" which supports incremental sigtar files.
* Merged in lp:~jnoster/duplicity/dpbx-added
- Add Dropbox backend
- NB! In order to use the backend one must:
1. Install Dropbox Python SDK first.
2. Run the duplicity with Dropbox backend (dpbx://) first time
* Merged in lp:~ed.so/duplicity/verify.data
- add switch --compare-data, to selectively enable formerly always disabled
data comparison on verify runs
* Merged in lp:~tblue/duplicity/paramiko-1.10.0
- This fixes bug #1156746, making the Paramiko backend compatible with
Paramiko 1.10.0. It keeps compatibility with older Paramiko versions.
* Merged in lp:~townsend/duplicity/fix-1161599-2
- The fix in revno. 912 didn't take into account that the parameter "body"
passed into request() is overloaded, so when it was NULL or of a type other
than file, it would fail. This checks if "body" is of type "file" before
actually seek()'ing back to the beginning of the file.
* Merged in lp:~tblue/duplicity/paramiko-fix-delete-retry
- This fixes bug #1115715, which is really annoying. Basically it makes
using the Paramiko backend with the default settings impossible.
* Merged in lp:~juan-f/duplicity/progress
- From time ago, there are people asking for a progress bar estimation in duplicity.
There is even a script that circumvents the issue, getting info from the log so as
to estimate the progress status ( https:/
but does not give enough feedback and the estimation is rather plain.
- I have developed a set of heuristics that gather information from the deltas and
the transfer ratios of the backend so as to forecast % of progress, estimation of
remaining time and average speed, for both full and incremental backup uploads.
- The current implementation works for boto backend, but to port the other backends
to use this feature would be quite easy (we can discuss the details if interested).
- The algorithm is activated by the --progress command line flag, and will perform a
first-pass dry-run to collect evidence for all the deltas. Next it will trigger the
real upload, while a thread statistically estimates the ratio of changes and
compression for the data in/out, and uses these ratios to forecast time remaining
and % of completion.
- The progress data will be logged each 3 seconds, or the --progress-rate flag.
* Merged in lp:~jnoster/duplicity/dpbx-added
- The application key was approved as "production" one after some changes to the code
to suit the requirements of Dropbox team (the keys are now obfuscated, for instance).
* Applied blocksize.patch from https:/
- New option --max-blocksize (default 2048) to allow increasing delta blocksize.
* Applied duplicity-
- Don't try to delete an empty file list.
* Merged in lp:~scowcron/duplicity/ftp_password_pexpect
- Use common backend.Backend get_password() rather than _ssh_pexpect.py specific code.
* Merged in lp:~mhu-s/duplicity/swiftbackend
- This branch adds support for Swift, the OpenStack Object Storage service. See
https:/
* Merged in lp:~verb/duplicity/boto-gcs
- These patches add support for Google Cloud Storage via the boto backend.
- boto has supported GCS in interoperability mode for a few years now. This change
adds support by taking advantage of boto's storage_uri abstraction layer.
* Merged in lp:~ckornacker/duplicity/megacloud
- Add support for Mega (mega.co.nz) backend.
* Applied patch from Eric S Raymond to man page to fix markup problems.
* Merged in lp:~ed.so/duplicity/man.page
- update paramiko links
- add command parameters to synopsis
- add --compare-data
- some polishing and several improvements
Duplicity 0.6.21 Released
Written for Duplicity by Kenneth Loafman on 2013-01-23
New in v0.6.21 (2013/01/23)
-------
Enhancements:
* Merged in lp:~ed.so/duplicity/24syntaxfix
- fix python 2.4 vs 2.5 syntax error
* Merged in lp:~mterry/duplicity/u1-oauthlib
- As the Ubuntu packager for duplicity, I would prefer u1backend.py
used oauthlib instead of oauth. oauthlib is well maintained upstream
(unlike oauth), has a python3 port (for the future), and is in Ubuntu
main (so is oauth right now, but hopefully in the future we can drop
it to universe, in which case duplicity can't use it anymore).
* Merged in lp:~mterry/duplicity/delete-new-sig-in-cache
- In duplicity 0.6.20, we fixed bug 1031269. This means that we no longer
leave sig files on the remote location. Leaving sig files on the remote
location also caused a bug with deleting cache files. Code used to leave
remote new-sig but delete the locale cache new-sig; this meant that we would
keep downloadoing the new-sig all the time from remote. We had worked around
that by just not deleting the new-sig in the cache, which was sort of the
wrong side of that problem to tackle. Now that we handle the remote
new-sigs better (by deleting them), I don't think we need this code anymore.
Patch by az@debian.org.
* Merged in lp:~mterry/duplicity/u1-ascii-error
- Fix for u1backend unicode error. Patch by Paul Barker.
* Merged in lp:~satwell/duplicity/caching
- Add a cache for password and group lookups. This significantly improves
runtime with very large password and group configurations.
* Merged in lp:~ed.so/duplicity/manpage
- more formatting fixes, clarifications in sections EXAMPLES, FILE SELECTION
* Merged in lp:~ed.so/duplicity/lftp.netrc
- Allow .netrc auth for lftp backend
* Merged in lp:~mterry/duplicity/946988
- This fixes bug 946988 by not duplicating the checks for when we should ask
for the password (those same checks are done more correctly inside
get_
* Merged in lp:~lenharo-h/duplicity/duplicity
- Generate encrypted backups without revealing the user's key id
via option --hidden-
* Merged in lp:~mterry/duplicity/u1-utf8
- Make sure u1backend returns filenames as utf8
* Merged in lp:~carlos-abalde/duplicity/gdocs-backend-gdata-2.0.16.-upgrade
- Upgrade of GoogleDocs backend to python gdata lib >= 2.0.15:
Stop using get_everything method.
* Merged in lp:~ed.so/duplicity/webdav.fix-retry
- bugfix: webdav retrying broke on ERRORS like "error: [Errno 32] Broken pipe" in
socket.pyas reported here https:/
added a more generalized 'retry_fatal' decorator which makes retrying backend
methods even easier
* Merged in lp:~ed.so/duplicity/manpage
- Clear up PASSPHRASE reusage as sign passphrase. Minor fixes.
* Merged in lp:~ed.so/duplicity/u1_and_manpage
- Manpage
- document Ubuntu One required python libs
- added continuous contributors and backend author notes
- U1backend
- lazily import non standard python libs, fixes
http://
- fix "not bytearray" prevents PUT with python 2.6
- don't hang after putting in credentials (cause it silently retries in background)
but go through with backup
* Fixed 1091269 Data corruption when resuming with --no-encryption
- Patches from Pascual Abellan that make block size consistent and
that add -n (no-encryption) option to manual-
- Modified gpg.py patch to use 64k block size so unit test passes.
* Merged in lp:~mterry/duplicity/static-corruption
- This branch fixes three possible ways a backup could get data-corrupted.
Inspired by bug 1091269.
A) If resuming after a volume that ended in a one-block file, we would
skip the first block of the next file.
B) If resuming after a volume that ended in a multi-block file, we would
skip the first block of the next file.
C) If resuming after a volume that spanned a multi-block file, we would
skip some data inside the file.
- A and B are because when finding the right place in the source files to
restart the backup, the iteration loop didn't handle None block numbers
very well (which are used to indicate the end of a file).
- C is what bug 1091269 talks about. This was because data block sizes would
get smaller as the difftar file got closer and closer to the volsize.
Standard block sizes were 64 * 1024. But say we were close to the end of
the difftar... When resuming, duplicity doesn't know the custom block sizes
used by the previous run, so it uses standard block sizes. And it doesn't
always match up, as you can imagine. So we would leave chunks of data out
of the backed up file.
- Tests added for these cases.
- This branch is called 'static-corruption' because all these issues occur
even when the source data doesn't change. I still think there are some
corruption issues when a file changes in between duplicity runs. I haven't
started looking into that yet, but that's next on my list.
- C only happened without encryption (because the gpg writer function already
happened to force a constant data block size). A and B happened with or
without encryption.
* Merged in lp:~ed.so/duplicity/webdav.fix-retry
- added ssl certificate verification (see man page)
- more robust retry routine to survive ssl errors, broken pipe errors
- added http redirect support
* Merged in lp:~ed.so/duplicity/webdav.manpage
- explanation of webdav changes above
* Merged in lp:~mterry/duplicity/pygi
- Python bindings for the gobject stack (used in the gio backend) have changed
from static to dynamically-
deprecated. So here's a branch to change the gio backend from old to new ones.
* Merged in lp:~mterry/duplicity/py3rsync
- This branch lets one build the _librsync module with Python 3. You can't
really do anything useful with it, but it's a nicely-isolated piece to add
Python 3 support for.
- The changes are a mix of modernization and #ifdef logic.
- All tests still pass in Python 2.7 and 2.4. I tested manually that the module
worked as expected in Python 3.
* Merged in lp:~duplicity-team/duplicity/po-updates
- Updated translations
Duplicity 0.6.20 Released
Written for Duplicity by Kenneth Loafman on 2012-10-28
New in v0.6.20 (2012/10/28)
-------
Enhancements:
* Merged in lp:~ed.so/duplicity/ssh.manpage
- added gdocs and rsync REQUIREMENTS
- added cloudfiles documentation
* Merged in lp:~ed.so/duplicity/gpginterface
- refactor GnuPGInterface to gpginterface.py
reasoning can be found in README
* Merged in lp:~ed.so/duplicity/duplicity.helpfix
- fix rare 'TypeError: encode() argument 1 must be string, not None'
- http://
* Merged in lp:~ed.so/duplicity/duplicity.tmpspacefix
- use tempfile.
- propbably solve bug 'Out of space error while restoring a file'
- https:/
- http://
* Merged in lp:~mterry/duplicity/utf8-po
- For some crazy reason, the gettext module defaults to giving you strings in
whatever charset the po file happened to define. Which means you never know
what string of bytes you're going to get. This module makes sure we always
get utf-8 byte strings. So we're at least predictable and reduces one
source of UnicodeDecodeErrors (like in bug 989496)
* Merged in lp:~mterry/duplicity/1031277
- ssh: actually delete all the requested files, not just the first one
* Merged in lp:~mterry/duplicity/leftover-sigtar
- So currently, duplicity does not delete signature files when doing a
remove-
useless and take up space.
- This branch does several things:
1) Make remove-all-but-n operate on chains. In practice it did before, since
the sets it operated on always came from complete chains (i.e. it never
used only some of the sets from a chain)
2) Add a new method to get all signature chains before a certain time.
3) Use this new method to also delete signature chains during remove-all-but
operations.
- And it cleans up the cleanuptest.py file:
1) Removes crufty, unused code
2) Disallows changing the destination folder for the test, which no one
would ever want to do and isn't really supported anyway
3) Add some additional checks to the existing test
4) Adds two new methods to test remove-all-but-n and
* Merged in lp:~ed.so/duplicity/duplicity.manpage
- disabled hyphenation and block justification for better readablility of
command line examples.
- reformatted REQUIREMENTS section for hopefully better online rendering
- minor clarifications
* Merged in lp:~gregretkowski/duplicity/cf-retry-delete
- This will retry cloudfile delete commands. With large numbers of archive
files over mediocre links transient network errors will occasionally cause
deletes to fail and these should be retried.
* Merged in lp:~mterry/duplicity/ropath.index
- This branch does two main things:
1) Skips base dir entries when compiling the list of deleted delta iters.
(this gracefully recovers from the sort of situations that lead to bug
929067). I'm reasonably confident this is an uninvasive change, but
please confirm.
2) Overwrites the sigtar file on backup-restart. This is because AFAICT,
duplicity will rewrite the entire sigtar each restart. But we were
opening the sigtar file as "ab", so we'd just dump the contents on top
of the previous contents. Which was causing some confusion in bug 929067.
If I'm wrong that we don't always rewrite the entire sigtar each time,
this needs some rethink. Please also confirm that.
- In addition, I added two tests for the above two changes and make some
improvements elsewhere in the restarttest.py file while I was at it.
* Merged in lp:~ed.so/duplicity/ssh-pexpect-msgbug
- Fixes 'UnboundLocalError: local variable 'msg' referenced before assignment'
in _ssh_pexpect.py
* Merged in lp:~ed.so/duplicity/gpg.tmp
- place gpg.py tempfiles in duplicity's tmp subfolder which is cleaned
whatever happens
* Merged in lp:~mterry/duplicity/u1-402
- Switch the code we check for out-of-space in u1backend.
* Applied patch for #1066625 ubuntu one backend
- add delay between retries
Bugs closed in this release:
519948 remove-* commands don't remove signature-files
907077 assert len(chain_list) == 2 AssertionError
929067 crash during restore: "assert ropath.index == (), ropath.index"
995851 doc improvement for --encrypt-key, --sign-key
997691 Ubuntu One backend should check for 402 error for out-of-space
1031269 remove-
1039001 --exclude-
with inaccessible other fs
1066625 ubuntu one backend does not work without gnome/dbus/x11 session
Duplicity 0.6.19 Released
Written for Duplicity by Kenneth Loafman on 2012-05-22
New in v0.6.19 (2012/05/22)
-------
Enhancements:
- lots of work on the man page to clean up requirements, etc.
- use empty listbody for enhanced webdav compatibility
- initial folder creation on backend does not result in a ResponseNotReady anymore
- add ssh_config support (/etc/ssh/
- add missing_host_key prompt to new sshbackend similar to ssh procedure
- added --ssh-backend parameter to switch between paramiko,pexpect
- allow answering gio mount questions (albeit naively)
- if the gio backend wants to ask a question during its mount phase, it previously just aborted.
- a couple more warning error codes that Deja Dup is interested in noticing.
- ssh paramiko backend respects --num-retries now
- set retry delay for ssh backends to 10s
- ssh pexpect backend
+ sftp part does not claim 'Invalid SSH password' although it's only 'Permission denied' now
+ sftp errors are now more talkative
- gpg.py
+ commented assert which broke otherwise working verify run
Bugs closed in this release:
588541 Connection failed, please check your password: Login dialog cancelled
Merges:
lp:~ed.so/duplicity/0.6-manpage
lp:~ed.so/duplicity/0.6-webdav_fixes
lp:~carlos-abalde/duplicity/gdocs-backend-gdata-2.0.16.-upgrade.
lp:~ed.so/duplicity/0.6-ssh_add_missinghostkey
lp:~ed.so/duplicity/0.6-readd_sshpexpect
Duplicity 0.6.18 Released
Written for Duplicity by Kenneth Loafman on 2012-02-29
New in v0.6.18 (2012/02/29)
-------
Enhancements:
- fix extraneous '.py' in botobackend.py include
- tests: add delay between backups to avoid assertion error
- tests: use backup source that is more likely to be larger than 1M compressed
- tests: make other-filesystem check more robust against certain directories being mounts or not
- resuming an incremental results in a 'Restarting backup, but current encryption settings do not match original settings' error because curtime is incorrectly set away from previous incremental value
- added option to not compress the backup, when no encryption is selected
- always delay a little bit when a backend gives us errors
- Don't cache TarInfo files. Tests still pass, so I don't believe we need the members cache (and in the old tarfile.py, we didn't cache either).
- Adding --file-prefix option so different sets of backups can be stored in the same bucket. See blueprint at https:/
- two changes that help the test suite pass
- raise log level on backend import failure so it will be visible under default conditions
- file /etc/motd may not exist in test environment. Use __file__ instead to point to a known plaintext source file.
- some code/import changes to make the ssh and boto backends compatible with Python 2.4.
- some changes to make roottest.py compatible with the new dir structure.
Bugs closed in this release:
884638 Python 2.5 / boto error
908228 possible memory leak
909031 SSH-Backend: Creating dirs separately causes a permissons-problems
916689 multipart upload fails on python 2.7.2
929465 UnsupportedBack
930727 ftpsbackend should respect num_retries for ftp commands
931175 duplicity crashes when PYTHONOPTIMIZE is set
Merges:
lp:~mterry/duplicity/always-delay
lp:~mterry/duplicity/memleak
lp:~mterry/duplicity/nopexpect
lp:~mterry/duplicity/resume-inc
lp:~mterry/duplicity/testfixes
lp:~nguyenqmai/duplicity/file-prefix-option
lp:~tobias-genannt/duplicity/nocompress
