deal with database files copied between machines and backup restoration

Bug #999574 reported by Eric Casteleijn
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
U1DB
Fix Released
High
Eric Casteleijn
u1db servers
Invalid
Undecided
John O'Brien

Bug Description

Replica ids need to be globally unique. When users copy database files between machines, things can break. We need to investigate whether we can and want to prevent this by identifying the machine the database is on, and including this information in the replica id, and verify it at runtime.

Tags: discussion

Related branches

Revision history for this message
Eric Casteleijn (thisfred) wrote :

 use ~/.u1db/databases to store the replica uids so you can move files around?

tags: added: discussion
John A Meinel (jameinel)
Changed in u1db:
importance: Undecided → High
status: New → Confirmed
milestone: none → pre-release
Revision history for this message
John A Meinel (jameinel) wrote :

The technique that seemed most feasible was to put most of the intelligence into sync. Specifically a few changes:

1) For every update to a document, instead of just storing the generation of the db, generate a unique identifier for each generation. (could be checksum based, but most likely just a UUID).

2) During sync of each document, we transmit the db generation + transaction id.

3) In the sync_log table, we store the gen of the other machine, along with the transaction id.

4) When we start a new sync, the get_sync_info call returns the known generation along with the known transaction id. We check if that information matches what we have stored. If it does, we continue syncing as normal.

5) If it does not, we fail.

6) We then provide a new api for "reset my replica_uid, and do a full sync".

Revision history for this message
John A Meinel (jameinel) wrote :

Note that this should handle the restore-from-backup case as well. Where a db gets reverted to a previous state in time, and we use it again for a bit, then do sync and the world burns. :)

One thing we haven't specifically worked through is if the SyncTarget might notice that it is getting an invalid data stream.

Specifically, you could get put_doc_if_newer with a generation + transaction id. The generation could be older than what you have stored as the most recent generation. Or it could match, but the transaction id is different.

However, the rollback-but-not-used case would like to not have to do a full sync. So instead, you could have the logic be:

If you get a document generation that is older than what we thought was last, *and* that content would be an insert, abort. If it is a conflict, we may have treated it as a conflict the last time we saw it. If it is converged, it is the same content. If it is superseded, we probably saw it in the past and just ignored it.

So the check becomes:

1) If the generation is newer than what we have, we are happy.
2) If the generation is equal to what we have, the transaction id must match or we fail the sync.
3) If the generation is older than what we have, the content must not be newer than what we have, or we fail the sync.

Revision history for this message
John A Meinel (jameinel) wrote :

We went this route instead of detecting copies, because it also handles the restore from backup case, which would otherwise have the exact same path on the exact same machine.

summary: - deal with database files copied between machines
+ deal with database files copied between machines and backup restoration
Changed in u1db:
status: Confirmed → Triaged
Revision history for this message
John A Meinel (jameinel) wrote :

Small addendum. We can treat existing data as just having a transaction id of the empty string, so that we don't have to regenerate all content from scratch.

Revision history for this message
John A Meinel (jameinel) wrote :

I'm splitting this bug into many smaller bugs, because it is a *huge* chunk of work to get all of this together.

So far the split out bugs are:
bug #1006870
bug #1006872
bug #1006876
bug #1006879
bug #1006882
bug #1006889

Rick McBride (rmcbride)
Changed in u1db:
assignee: nobody → Eric Casteleijn (thisfred)
Changed in u1db-servers:
assignee: nobody → Eric Casteleijn (thisfred)
Rick McBride (rmcbride)
Changed in u1db-servers:
status: New → Confirmed
Changed in u1db:
status: Triaged → In Progress
Changed in u1db:
status: In Progress → Fix Committed
Changed in u1db:
milestone: pre-release → 0.0.3
status: Fix Committed → Fix Released
Revision history for this message
John O'Brien (jdobrien) wrote :

Fix exception propagation in engine

Changed in u1db-servers:
assignee: Eric Casteleijn (thisfred) → John O'Brien (jdobrien)
status: Confirmed → Invalid
Revision history for this message
John O'Brien (jdobrien) wrote :

Copy/paste FAIL. This was marked invalid because it can't really happen with the servers.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.