News and announcements

Duplicity 0.7.0 Released

Written for Duplicity by Kenneth Loafman on 2014-10-23

This is the first release of the 0.7 series. Since it is a dot zero release, you know what to expect.

New in v0.7.00 (2014/10/23)
---------------------------
Enhancements:
* Adjust unit tests to expect single FTP backend
* Merged in lp:~moritzm/duplicity/duplicity
  - Use lftp for both FTP and FTPS
* Merged in lp:~ed.so/duplicity/0.7-dpbx.importfix
  - fix this showstopper with the dropbox backend
    "NameError: global name 'rest' is not defined"
* Merged in lp:~jflaker/duplicity/BugFix1325215
  - The reference to "--progress_rate" in the man page as a parameter is
    incorrect. Should be "--progress-rate".
* Merged in lp:~hooloovoo/duplicity/updated-README-REPO
  - Changes to README-REPO to reflect the restructuring of the directories.
* Fixed bug 1375304 with patch supplied by Aleksandar Ivanovic
* Merged in lp:~ed.so/duplicity/webdav200fix-0.7
  - webdav backend fix "BackendException: Bad status code 200 reason OK. " when
    restarting an interrupted backup and overwriting partially uploaded volumes.
* Merged in lp:~mterry/duplicity/require-2.6
  - Require at least Python 2.6.
  - Our code base already requires 2.6, because 2.6-isms have crept in. Usually
    because we or a contributor didn't think to test with 2.4. And frankly,
    I'm not even sure how to test with 2.4 on a modern system.
* Merged in lp:~mterry/duplicity/drop-pexpect
  - Drop our local copy of pexpect in favor of a system version.
  - It's only used by the pexpect ssh backend (and if you're opting into that,
    you probably can expect that you will need pexpect) and the tests.
  - I've done a quick smoketest (backed up and restored using
    --ssh-backend=pexpect) and it seemed to work fine with a modern version
    of pexpect.
* Merged in lp:~mterry/duplicity/2.6isms
  - Here's a whole stack of minor syntax modernizations that will become
    necessary in python3. They all work in python2.6.
  - I've added a new test to keep us honest and prevent backsliding on these
    modernizations. It runs 2to3 and will fail the test if 2to3 finds anything
    that needs fixing (with a specific set of exceptions carved out).
  - This branch has most of the easy 2to3 fixes, the ones with obvious and
    safe syntax changes.
  - We could just let 2to3 do them for us, but ideally we use 2to3 as little
    as possible, since it doesn't always know how to solve a given problem.
    I will propose a branch later that actually does use 2to3 to generate
    python3 versions of duplicity if they are requested. But this is a first
    step to clean up the code base.
* Merged in lp:~mterry/duplicity/drop-static
  - Drop static.py.
  - This is some of the oldest code in duplicity! A bzr blame says it is
    unmodified (except for whitespace / comment changes) since revision 1.
  - But it's not needed anymore. Not really even since we updated to python2.4,
    which introduced the @staticmethod decorator. So this branch drops it and
    its test file.
* Merged in lp:~mterry/duplicity/py3-map-filter
  - In py3, map and filter return iterable objects, not lists. So in each case
    we use them, I've either imported the future version or switched to a list
    comprehension if we really wanted a list.
* Merged in lp:~mterry/duplicity/backend-unification
  - Reorganize and simplify backend code. Specifically:
    - Formalize the expected API between backends and duplicity. See the new
      file duplicity/backends/README for the instructions I've given authors.
    - Add some tests for our backend wrapper class as well as some tests for
      individual backends. For several backends that have some commands do all
      the heavy lifting (hsi, tahoe, ftp), I've added fake little mock commands
      so that we can test them locally. This doesn't truly test our integration
      with those commands, but at least lets us test the backend glue code.
    - Removed a lot of duplicate and unused code which backends were using (or
      not using). This branch drops 700 lines of code (~20%)
      in duplicity/backends!
    - Simplified expectations of backends. Our wrapper code now does all the
      retrying, and all the exception handling. Backends can 'fire and forget'
      trusting our wrappers to give the user a reasonable error message.
      Obviously, backends can also add more details and make nicer error
      messages. But they don't *have* to.
    - Separate out the backend classes from our wrapper class. Now there is no
      possibility of namespace collision. All our API methods use one
      underscore. Anything else (zero or two underscores) are for the backend
      class's use.
    - Added the concept of a 'backend prefix' which is used by par2 and gio
      backends to provide generic support for "schema+" in urls -- like par2+
      or gio+. I've since marked the '--gio' flag as deprecated, in favor of
      'gio+'. Now you can even nest such backends like
      par2+gio+file://blah/blah.
    - The switch to control which cloudfiles backend had a typo. I fixed this,
      but I'm not sure I should have? If we haven't had complaints, maybe we
      can just drop the old backend.
    - I manually tested all the backends we have (except hsi and tahoe -- but
      those are simple wrappers around commands and I did test those via mocks
      per above). I also added a bunch more manual backend tests to
      ./testing/manual/backendtest.py, which can now be run like the above to
      test all the files you have configured in config.py or you can pass it a
      URL which it will use for testing (useful for backend authors).
* Merged in lp:~mterry/duplicity/py2.6.0
  - Support python 2.6.0.
  - Without this branch, we only support python >= 2.6.5 because that's when
    python's urlparse.py module became its more modern incarnation. (I won't
    get into the wisdom of them making such a change in the middle of the
    2.6 lifecycle.)
  - Also, the version of lockfile that I have (0.8) doesn't work with python
    2.6.0 or 2.6.1 due to their implementation of
    threading.current_thread().ident returning None unexpectedly. So this
    branch tells lockfile not to worry about adding the current thread's
    identifier to the lock filename (we don't need a separate lock per thread,
    since our locking is per process).
  - I've tested with 2.6.0 and 2.7.6 (both extremes of our current support).
* Update shebang line to python2 instead of python to avoid confusion.
* Merged in lp:~3v1n0/duplicity/copy.com-backend
  - I've added a backend for Copy.com cloud storage, this supports all the
    required operations and works as it should from my tests.
  - You can use it by calling duplicity with something like:
    copy://account@email.com:your-password@copy.com/duplicity
  - The only thing I've concerns with is the optimized support for _delete_list
    which can't be enabled here because the test_delete_list tries also to
    delete a not-existing files, and it requires the backend not to raise an
    exception in that case (is this somewhat wanted or could we do the same as
    for _delete or _query?)
* Merged in lp:~ed.so/duplicity/webdav200fix-0.7
  - webdav backend fix "BackendException: Bad status code 200 reason OK. " when
    restarting an interrupted backup and overwriting partially uploaded volumes.
* Merged in lp:~mterry/duplicity/webdav-fixes
  - This branch fixes two issues I saw when testing the webdav backend:
  - 1) Errors like the following: "Attempt 1 failed. BackendException: File
    /tmp/duplicity-LQ1a0i-tempdir/mktemp-u2aiyX-2 not found locally after get
    from backend". These were caused by the _get() method not calling setdata()
    on the local path object, so the rest of the code thought it didn't exist.
  - 2) Some odd issues from stale responses/data. We have a couple places in
    webdavbackend.py where we close the connection before making a request
    because of this problem. But I've changed it to do it every time, more
    reliably, by putting a _close() call inside the request() method.
  - With this, the webdav backend seems fine to me.
* Merged in lp:~antmak/duplicity/0.7-par2-fix
  - Useful fix for verbatim par2cmdline options (like "-t" in par2-tbb version)
* Fixed bug 1327550: OverflowError: signed integer is greater than maximum
  - Major and minor device numbers are supposed to be one byte each. Someone
    has crafted a special system image using OpenVZ where the major and minor
    device numbers are much larger (ploop devices). We treat them as (0,0).
* Added sxbacked.py, Skylable backend. Waiting on man page updates.
* Merged in lp:~ed.so/duplicity/manpage.verify
  - Clarify verify's functionality as wished for by a user surprised with a big
    bandwidth bill from rackspace.
* Merged in lp:~jeffreydavidrogers/duplicity/duplicity
  - This change fixes two small typos in the duplicity man page.
* Merged in lp:~johnleach/duplicity/1315437-swift-container-create
  - Check to see if the swift container exists before trying to create it,
    in case we don't have permissions to create containers. Fixes #1315437
* Merged in lp:~ed.so/duplicity/manpage.blocksize
  - add --max_blocksize doc
  - reorder 'a note on filename prefixes' into alphabetical order

Duplicity 0.6.24 Released

Written for Duplicity by Kenneth Loafman on 2014-10-20

New in v0.6.25 (2014/10/20)
---------------------------
Enhancements:
* Add deprecation warning for 0.6 series
* Merged in lp:~ed.so/duplicity/0.6-dpbx.importfix
  - fix this showstopper with the dropbox backend
    "NameError: global name 'rest' is not defined"
* Merged in lp:~jflaker/duplicity/BugFix1325215
  - The reference to "--progress_rate" in the man page as a parameter is
    incorrect. Should be "--progress-rate".
* Merged in lp:~hooloovoo/duplicity/updated-README-REPO
  - Changes to README-REPO to reflect the restructuring of the directories.
* Fixed bug 1375304 with patch supplied by Aleksandar Ivanovic
* Merged in lp:~ed.so/duplicity/webdav200fix-0.6
  - webdav backend fix "BackendException: Bad status code 200 reason OK. " when
    restarting an interrupted backup and overwriting partially uploaded volumes.
* Merged in lp:~antmak/duplicity/0.6-par2-fix
  - Useful fix for verbatim par2cmdline options (like "-t" in par2-tbb version)
* Merged in lp:~jon-haggblad/duplicity/ftps-fix
  - Minor bugfix for ftps backend where lftp needs host prefixed by "ftps://".
* Fixed bug 1327550: OverflowError: signed integer is greater than maximum
  - Major and minor device numbers are supposed to be one byte each. Someone
    has crafted a special system image using OpenVZ where the major and minor
    device numbers are much larger (ploop devices). We treat them as (0,0).
* Merged in lp:~jon-haggblad/duplicity/ftps-fix (2nd try)
  - Minor bugfix for ftps backend where lftp needs host prefixed by "ftps://".
* Merged in lp:~johnleach/duplicity/1315437-swift-container-create
  - Check to see if the swift container exists before trying to create it,
    in case we don't have permissions to create containers. Fixes #1315437

Please NOTE: The 0.6 series is in the process of being deprecated and the 0.6.25 release is the last release of the series that will receive new additions or enhancements. It will continue to receive critical bug fixes to support distributions with long term support. The major focus of development will be on the 0.7 series and the first of that series will be released shortly.

Duplicity 0.6.24 Released

Written for Duplicity by Kenneth Loafman on 2014-05-09

New in v0.6.24 (2014/05/09)
---------------------------
Enhancements:
* Applied two patches from mailing list message at:
  https://lists.nongnu.org/archive/html/duplicity-talk/2014-01/msg00030.html
  "Added command line options to use different prefixes for duplicity files"
  This resolves https://bugs.launchpad.net/duplicity/+bug/1170161 and provides
  a workaround for https://bugs.launchpad.net/duplicity/+bug/1170113
* Merged in lp:~mterry/duplicity/gpg-encode
  - getpass.getpass(prompt) eventually calls str(prompt). Which is a no go,
    if the prompt contains unicode. Here's a patch to always pass getpass() a
    byte string.
  - Our tests didn't catch this because they always set PASSPHRASE. I've added
    a test that passes the passphrase via stdin.
* Merged in lp:~mterry/duplicity/pexpect-fix
  - duplicity has its own copy of pexpect. Use that instead of requiring one
    from the system.
* Merged in lp:~prateek/duplicity/s3-glacier
  - Fixes https://bugs.launchpad.net/duplicity/+bug/1039511
    - Adds support to detect when a file is on Glacier and initiates a restore
      to S3. Also merges overlapping code in the boto backends
  - Fixes https://bugs.launchpad.net/duplicity/+bug/1243246
    - Adds a --s3_multipart_max_timeout input option to limit the max execution
      time of a chunked upload to S3. Also adds debug message to calculate
      upload speed.
* Merged in lp:~ed.so/duplicity/fix.dpbx
  - Fix dpbx backend "NameError: global name 'rest' is not defined"
* Merged in lp:~prateek/duplicity/botoimportfix
  - Switches the boto backend back to using lazy imports so there are no
    complaints during the importing of backends.
* Merged in lp:~germer/duplicity/par2
  - This branch adds Par2 recovery files to duplicity. It is a wrapper backend
    which will create the recovery files and upload them all together with the
    wrapped backend. Corrupt archives will be detected and repaired (if
    possible) on the fly during restore.
  - It can be used with url-string par2+webdavs://USER@HOST/PATH
  - Fixes https://bugs.launchpad.net/duplicity/+bug/426282
* Merged in lp:~fredrik-loch/duplicity/duplicity-S3-SSE
  - Adds support for server side encryption as requested in Bug #996660
* Merged in lp:~mterry/duplicity/modern-testing
  - Enable/use more modern testing tools like nosetests and tox as well as more
    common setup.py hooks like test and sdist.
  - Specifically:
    * move setup.py to toplevel where most tools and users expect it
    * Move and adjust test files to work when running "nosetests" in toplevel
      directory. Specifically, do a lot more of the test setup in
      tests/__init__.py rather than the run-tests scripts
    * Add small tox.ini file for using tox, which is a wrapper for using
      virtualenv. Only enable for py26 and py27 right now, since modern
      setuptools dropped support for <2.6 (and tox 1.7 recently dropped <2.6)
    * Add setup.py hooks for test and sdist which are both standard targets
      (sdist just outsources to dist/makedist right now)
* Merged in lp:~mterry/duplicity/drop-u1
  - Ubuntu One is closing shop. So no need to support a u1 backend anymore.
* Merged in lp:~mterry/duplicity/fix-drop-u1
  - Looks like when the drop-u1 branch got merged, its conflict got resolved
    badly. Here is the right version of backend.py to use (and also drops
    u1backend.py from POTFILES).
* Merged in lp:~mterry/duplicity/encode-for-print
  - Encode translated strings before passing them to 'print'.
  - The print command can only apparently handle bytes. So when we pass it
    unicode, it freaks out. There were only four instances I saw where we used
    print, so I figured it was easiest to just convert them to use the log
    framework too.
  - That way all user-visible strings go through that framework and are subject
    to the same encoding rules.
* Merged in lp:~mterry/duplicity/more-test-reorg
  - Here's another test reorganization / modernization branch. It does the
    following things:
    - Drop duplicity/misc.py. It is confusing to have both misc.py and util.py,
      and most of the code in misc.py was no longer used. I moved the one
      function that was still used into util.py.
    - Consolidated the various ways to run tests into just one. I made tox runs
      go through ./setup.py test, rather than nosetests. And I made the
      ./testing/run-tests scripts just call tox. Now we no longer need nosetests
      as a test dependency (although you can still use it if you want).
    - Added two more code quality automated tests: a pep8 one and a pylint one.
      I disabled almost all checks in each program that gave a warning. These
      tests just establish a baseline for future improvement.
    - Moved the test helper code into TestCase subclasses that all tests can
      use. And used more code sharing and setUp/tearDown cleverness to remove
      duplicated code.
    - Reorganized the tests in ./testing/tests into ./testing/functional and
      ./testing/unit -- for whether they drive duplicity as a subprocess or
      whether they import and test code directly. Each dir can have specialized
      TestCase subclasses now.
    - Renamed the files in ./testing/unit to more clearly indicate which file
      in ./duplicity they are unit testing.
    - Added some helper methods for tests to set environment and globals.*
      parameters more safely (i.e. without affecting other tests) by
      automatically cleaning up any such changes during test tearDown.
    - Removed test_unicode.py, since it is kind of dumb. It used to be more
      useful, but now with py2.6, we are just testing that one line of code
      in it is actually there.
* Fixed bug #1312328 WebDAV backend can't understand 200 OK response to DELETE
  - Allow both 200 and 204 as valid response to delete
* Merged in lp:~mterry/duplicity/encode-exceptions
  - Because exceptions often contain file paths, they have the same problem
    with Python 2.x's implicit decoding using the 'ascii' encoding that we've
    experienced before. So I added a new util.uexc() method that uses the
    util.ufn() method to convert an exception to a unicode string and used it
    around the place.
  - Bugs fixed: 1289288, 1311176, 1313966
* Applied expat fix from edso. See answer #12 in
  https://answers.launchpad.net/duplicity/+question/248020

Duplicity 0.6.23 Released

Written for Duplicity by Kenneth Loafman on 2014-01-24

New in v0.6.23 (2014/01/24)
---------------------------
Enhancements:
* Applied patch from bug 1216921 to fix ignore_missing().
  - merged lp:~mterry/duplicity/ignore-missing to fix patch.
* Merged in lp:~mterry/duplicity/catch-seq-copy-error
  - Any* exception when running patch_seq2ropath should be ignored (though
    logged) and duplicity should move on. This covers the two asserts in that
    function (bug 1155345 and bug 720525) as well as errors that happen during
    file copying (bug 662442).
* Merged in lp:~mterry/duplicity/argv
  - Fix use of argv when calling os.execve
* Merged in lp:~verb/duplicity/bucket_root_fix
  - Fix bug that prevents backing up to the root of a bucket with boto backend.
* Merged in lp:~gliptak/duplicity/415619
  - Better error message when chown fails
* Merged in lp:~mterry/duplicity/log-path-type
  - Any backup browser built on top of duplicity will need to indicate which
    files in the backup are folders and which are files. The current logging
    information doesn't provide this detail. So I've added a field to the
    log.InfoCode.file_list output that includes the path type.
* Merged in lp:~mterry/duplicity/manifest-oddities
  - We may accidentally end up with an oddly inconsistent manifest like so:
    Volume 1
    Volume 2
    Volume 3
    Volume 2
    As did get reported recently on the mailing list:
    http://lists.nongnu.org/archive/html/duplicity-talk/2013-11/msg00009.html
  - One way this can happen (the only way?) is if you back up, then duplicity
    gets interrupted between writing the manifest and uploading the volume.
    Then, when restarted, there is no longer enough data to create as many
    volumes as existed previously.
  - This situation can cause an exception when trying to restart the backup.
  - This branch fixes it by deleting any excess volume information encountered
    when loading in the manifest. We discard volume with higher numbers
    than the last one read.
* Merged in lp:~mterry/duplicity/disappearing-source
  - When restarting a backup, we may accidentally skip the first chunk of one of
    the source files. To reproduce this,:
    1) interrupt a backup
    2) delete the source file it was in the middle of
    3) restart the backup
  - When replaying the source iterator to find where to resume from, we can't
    notice that the file is gone until we've already iterated past where it
    would be!
  - The solution I came up with is to just let duplicity stuff the data we
    accidentally read back into the source iterator.
  - This is actually a data loss bug, because it's possible to back up
    corrupted files (that are missing their first chunk).
* Merged in lp:~mterry/duplicity/normalize-before-using
  - Avoid throwing an exception due to a None element in a patch sequence.
  - None elements in a (non-normalized) patch sequence are perfectly normal.
    With the current code in the patched function, it is certainly possible to
    hit a crash due a None.
    See http://lists.nongnu.org/archive/html/duplicity-talk/2013-11/msg00005.html
  - This branch fixes that by normalizing the sequence before using it in the
    logging code. It's acceptable to bring the normalize_ps() call outside the
    try/except block because normalize_ps is not expected to throw. It's
    relatively simple and doesn't really use its objects besides checking if
    they are None.
* Applied patch to fix "Access GDrive through gdocs backend failing"
  - see https://lists.nongnu.org/archive/html/duplicity-talk/2013-07/msg00007.html
* Merged in lp:~jkrauss/duplicity/pyrax
  - Rackspace has deprecated python-cloudfiles in favor of their pyrax
    library, which consolidates all Rackspace Cloud API functionality into
    a single library. Tested it with Duplicity 0.6.21 on both Arch Linux
    and FreeBSD 8.3.0.
* Changed to default to pyrax backend rather than cloudfiles backend.
  To revert to the cloudfiles backend use '--cf-backend=cloudfiles'
* Merged in lp:~verb/duplicity/boto-min-version
  - Update documentation and error messages to match the current actual version
    requirements of boto backend.
* Merged in lp:~ed.so/duplicity/debian.paramiko.log
  - upstream debian patch "paramiko logging"
    http://patch-tracker.debian.org/package/duplicity/0.6.22-2
* Merged in lp:~ed.so/duplicity/debian.dav.mkdir
  - upstream debian patch "webdav create folder recursively"
    http://patch-tracker.debian.org/package/duplicity/0.6.22-2
* Nuke tabs
* Merged in lp:~mterry/duplicity/encoding
  - This branch hopefully fixes two filename encoding issues:
  - Users in bug 989496 were noticing a UnicodeEncodeError exception which
    happens (as far as I can tell) because some backends (like webdav) are
    returning unicode filenames from list(). When these filenames are combined
    with the utf8 translations of log messages, either (A) the default ascii
    encoding can't handle promoting the utf8 bytes or -- if there aren't any
    utf8 bytes in the translation -- (B) the resulting unicode string raises
    an error later when log.py tries to upgrade the string again to unicode
    for printing.
  - This fix is largely implemented by adding a wrapper for backend list()
    implementations. This wrapper ensures that duplicity internals always see
    a byte string. (I'd like to eventually use this same wrapping strategy to
    implement generic retry support without backends having to add any logic,
    but that's just a thought for the future.)
  - That is, the fix for issue #1 is completely inside backend.py and the
    changes to backends/*.py.
  - The rest of the invasive changes deal with filenames that may not be valid
    utf8. This is much rarer, but possible. For proper handling of this, we
    need to print using unicode, and convert filenames from the system filename
    encoding to unicode, gracefully handling conversion errors. Some of the
    filenames we print are remote names. Who knows what encoding they are in;
    it could be different than the system filename encoding. 99% of the time,
    everything will be utf8 and we're fine. If we do get conversion errors,
    the only effect should be some question mark characters in duplicity
    logging output.
  - I tried to convert as much of the actual codebase to use unicode for
    printing. But I stopped short of adding an assert in log.py to enforce
    unicode, because I didn't want to go through all the backend code and
    manually adjust those bits without being able to test each one.
* Restored missing line from patch of gdocsbackend.py
* Reverted changes to gdocsbackend.py
* Restored patch of gdocsbackend.py from original author (thanks ede)
* Applied patch from bug 1266753: Boto backend removes local cache if
  connection cannot be made
* Merged in lp:~louis-bouchard/duplicity/add-allow-concurrency
  - Implement locking mechanism to avoid concurrent execution under the same
    cache directory. This is the default behavior.
  - Also implement --alllow-concurrency option to disable the locking
    if required.
  - This functionality adds a dependency to python-lockfile