News and announcements

Dropping support for V0 protocol and schema

Written for FileStore by Jason Gerard DeRose on 2015-11-22

The V1 Dmedia hashing protocol, V1 Dmedia document schema, and V1 Novacut document schema were finalized over 29 months ago (on June 1st 2013):

My feeling is that this has given folks ample time to migrate any existing V0 libraries to V1. However, if anyone out there has an existing V0 library and needs help migrating it to V1, please let me know and I'll happily help!

Still, there is a cost in maintaining the V0 to V1 migration path, especially considering some refactoring I'd like to do in the near future. The V0 to V1 migration was unusual in terms of what we expect going forward, and the code for it is rather crufty.

So this migration path has now been dropped in our daily PPA builds for Ubuntu 15.10/Wily and Ubuntu 16.04/Xenial.

Note that support for this migration path is still present in our daily *and* stable PPA builds for Ubuntu 14.04/Trusty and Ubuntu 15.04/Vivid, and that support is still present in our *stable* PPA for Ubuntu 15.10/Wily. The current stable release for 15.10/Wily will be archived in a separate PPA prior to releasing a new stable release for 15.10/Wily that drops support for this migration path.

At this time, no further stable releases are planned for Ubuntu 14.04/Trusty or Ubuntu 15.04/Vivid, so Novacut releases that support the V0 to V1 migration path will be available for as long as each respective Ubuntu release is supported (almost 3 years more for Ubuntu 14.04/Trusty, and about 2 months more for Ubuntu 15.04/Vivid).

This is all part of an effort to trim some fat in preparation for a (hopefully) exciting Novacut release for Ubuntu 16.04/Xenial, the next Ubuntu LTS release.

Finally, note the plan is to indefinitely support the V1 hashing protocol, and to indefinitely support migration from the V1 document schema to whatever document schema version Dmedia and Novacut currently use.

Read more

Updated docs, Protocol Design Rationale

Written for FileStore by Jason Gerard DeRose on 2014-05-19

Big update to the FileStore documentation this month. In particular, you might want to check out the revamped Protocol Design Rationale (which originally just documented the V0 interim protocol):

Read more

V1 hashing protocol is go!

Written for FileStore by Jason Gerard DeRose on 2013-06-01

So big news! The Version 1 Dmedia Hashing Protocol has been finalized, and is
now used by default.

The only downside is that because of the switch to our Dbase32 encoding, it
wasn't possible to support V0 alongside V1. The first time you load a FileStore
containing a V0 files/ layout, it will move this directory to files0/,
preserving your data.

But to actually use these files, you must run this upgrade script from a


Or optionally, if you only have Dmedia installed, run this (which the above
calls before upgrading the Novacut databases):


For more help, please watch this screencast that walks you through the upgrade

There will certainly be additional protocols added in the future, but from V1
forward, all protocols will be supported indefinitely. Newly added files will
be hashed according to whatever the newest protocol version is at the time, but
you will always be able to resolve and verify a file according to an ID computed
with an older protocol version.

This is important so that we can always preserve the link integrity between,
say, a Novacut edit and the files that edit references.

V1 is a big milestone not just because it finalizes the details of V1, but also
because it finalizes our protocol framework, the design aspects we expect to be
the same across all protocol versions from V1 onward.

Each new protocol version will use a different digest size, and we'll use the
ID length to determine what protocol version was used to compute an ID. So for
example, V1 uses a 240-bit digest (48 characters when Dbase32 encoded), and V2
might use a 280-bit digest (56 characters when Dbase32 encoded).

Things that will (or can) change between protocol versions:

    * By definition, the digest size will change

    * The underlying hash function likely will change

    * The leaf size might change

And things we don't expect to change:

    * The digest size will always be a multiple of 40 bits

    * The root-hash and leaf-hashes will use the same digest size

    * The leaf size will be a multiple of 2 MiB

    * The root-hash will be cryptographically tied to the file-size

    * The leaf-hashes will be cryptographically tied to the leaf-index

    * Dbase32 will always be the "official" digest encoding, and will be the
      encoding use by standard FileStore layouts

We want to start research on V2 soon, but we likewise want to be very leisurely
about finalizing V2 (which is why we should start research soon). I hope it
will be at least 5 years till we add another protocol.

If anyone is interested in such research, the Protocol base class makes it quite
easy to build experimental protocols:

Also, folks might be interested in reading the V1 protocol specification:

(Thanks again to Robert von Burg for working on an experimental Java
implementation, which provided lots of great feedback that helped make the
specification clearer and more correct.)

And one last note, just because people tend to ask a lot: I'll explain why we
stuck with Skein for V1, rather than using the SHA-3 winner (Keccak).

The biggest reason we stuck with Skein is performance, specifically performance
of 64-bit software implementations. For our current 240-bit digest size, Skein
is roughly twice as fast as Keccak (on my hardware anyway). And that is
significant, because even Skein can't quite keep up with the read throughput of
todays fastest SSDs when hashing on a single core.

Although Keccak can be faster than Skein when implemented in hardware, we have
to make decisions based on the here and now. Today, no such hardware
implementations are readily available, and it might be many years before they

Another issue is the Skein parameter system made our cryptographic tying very
easy, but HMAC (or equivalent) hasn't been defined yet for Keccak, and defining
our own would be very risky. Yet lots of practical experience working through
different protocol iterations, and building useful software on top, has made me
more convinced than ever that this cryptographic tying is an extremely important
and pragmatic feature for our use case.

And the last thing is I've embraced the idea that we will have additional
protocol versions sooner rather than later. We will support V1 forever, not use
it by default forever. I think it's important that we architect the `filestore`
package to support multiple protocol versions as soon as possible, long before
we actually have a second protocol. This is inevitable, so let's start
practicing now.

We really needed to commit to a stable protocol version *now*, and I think at
this moment Skein is the best choice. Plus, if we switched to a different hash
algorithm now, it would still take time to understand it as well as we
understand Skein, to be confident about the way in which we are using it. That
could easily mean another year till we have a stable protocol.

Anyway, thanks to the many people who have patiently reviewed the many protocol
iterations leading up to V1, especially David Jordan, Hagen F├╝rstenau, and
Robert von Burg.


Updated .

Formal protocol specification

Written for FileStore by Jason Gerard DeRose on 2013-01-28

As a step toward finalizing the version 1 Dmedia hashing protocol, we're preparing a more formal specification:

Thanks to Robert von Burg, we're already aware of a number of shortcomings in the specification, but we'd love feedback on our work-in-progress.

Read more

14 of 4 results