You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Damien Katz <da...@apache.org> on 2009/07/15 00:06:35 UTC

Attachment level replication

I've just finished the deterministic revs work, and I've made changes  
to the attachment meta data is stored. I figure, since I'm in there  
changes things, I should go ahead and make a change to allow for  
incremental attachment replication,so we only replicate the binary  
attachments (which can be very large) that have changed, instead of  
all the attachments any something in the document changes.

Currently when we replicate a changed document, we replicate all the  
attachments, regardless if they've changed.

It looks something like this:

Replicator get the revs of the latest changes since seq N from the  
source.
Replicator asks the target "do you have these revs?"
Target responds, "here are the revs I am missing"
Replicator asks source, give me these revs I am missing or latest  
revisions of those revs.
Source returns the docs and attachment info.
Replicator writes documents and attachments to target


For incremental replication, CouchDB would now track which revision a  
document was edited in, storing the revision number along with the  
attachment metadata. When an attachment is updated, the revision  
number is updated along with it. And since we hash the attachments  
contents, we can be smart an updated it's rev number it only when it  
actually changes.

New Replication:

Replicator get the revs of the latest changes since seq N from the  
source.
Replicator asks the target "do you have these revs?"
Target responds, "here are the revs I am missing, and these are the  
latest revs I do have"
Replicator asks source, give me these revs I am missing or latest  
revisions of those revs.
Source returns the docs and attachment info.
Replicator figures which, if any, earlier revisions of the doc already  
exist on the target, using the "latest revs" the target gave us.
Replicator writes to target the documents and only the attachments  
that have changed since the latest revisions already on target.

These would require some new apis in addition to changes in the  
replicator, but we can keep around the old apis so old versions can  
still replicate to us.

Feedback please.

-Damien

Re: Attachment level replication

Posted by Antony Blakey <an...@gmail.com>.
Excellent. Additionally, if the source and destination share an old  
version of the attachment, an xdelta/rsync diff would potentially save  
a lot of bandwidth.

Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787

The intuitive mind is a sacred gift and the rational mind is a  
faithful servant. We have created a society that honours the servant  
and has forgotten the gift.
   -- Albert Einstein



Re: Attachment level replication

Posted by Ben Browning <be...@gmail.com>.
>From what you describe, it sounds like an excellent idea.

Do you have a clear idea of what the new APIs would be and if they're
potentially useful outside of the new replicator?

Also, I don't think it was explicitly mentioned, but is this slated to
make it into a 0.10.x release?

Ben

Re: Attachment level replication

Posted by Adam Kocoloski <ko...@apache.org>.
On Jul 14, 2009, at 6:06 PM, Damien Katz wrote:

> I've just finished the deterministic revs work, and I've made  
> changes to the attachment meta data is stored. I figure, since I'm  
> in there changes things, I should go ahead and make a change to  
> allow for incremental attachment replication,so we only replicate  
> the binary attachments (which can be very large) that have changed,  
> instead of all the attachments any something in the document changes.
>
> Currently when we replicate a changed document, we replicate all the  
> attachments, regardless if they've changed.
>
> It looks something like this:
>
> Replicator get the revs of the latest changes since seq N from the  
> source.
> Replicator asks the target "do you have these revs?"
> Target responds, "here are the revs I am missing"
> Replicator asks source, give me these revs I am missing or latest  
> revisions of those revs.
> Source returns the docs and attachment info.
> Replicator writes documents and attachments to target
>
>
> For incremental replication, CouchDB would now track which revision  
> a document was edited in, storing the revision number along with the  
> attachment metadata. When an attachment is updated, the revision  
> number is updated along with it. And since we hash the attachments  
> contents, we can be smart an updated it's rev number it only when it  
> actually changes.
>
> New Replication:
>
> Replicator get the revs of the latest changes since seq N from the  
> source.
> Replicator asks the target "do you have these revs?"
> Target responds, "here are the revs I am missing, and these are the  
> latest revs I do have"
> Replicator asks source, give me these revs I am missing or latest  
> revisions of those revs.
> Source returns the docs and attachment info.
> Replicator figures which, if any, earlier revisions of the doc  
> already exist on the target, using the "latest revs" the target gave  
> us.
> Replicator writes to target the documents and only the attachments  
> that have changed since the latest revisions already on target.
>
> These would require some new apis in addition to changes in the  
> replicator, but we can keep around the old apis so old versions can  
> still replicate to us.
>
> Feedback please.
>
> -Damien

Sounds good.  I think we shouldn't have any trouble preserving  
bidirectional replication with 0.9.x, either.  Best,

Adam 

Re: Attachment level replication

Posted by Zachary Zolton <za...@gmail.com>.
Will 0.10 be replication compatible with 0.9?

Perhaps we could keep a compatibility matrix in the wiki...

On Jul 14, 2009, at 5:06 PM, Damien Katz <da...@apache.org> wrote:

> I've just finished the deterministic revs work, and I've made  
> changes to the attachment meta data is stored. I figure, since I'm  
> in there changes things, I should go ahead and make a change to  
> allow for incremental attachment replication,so we only replicate  
> the binary attachments (which can be very large) that have changed,  
> instead of all the attachments any something in the document changes.
>
> Currently when we replicate a changed document, we replicate all the  
> attachments, regardless if they've changed.
>
> It looks something like this:
>
> Replicator get the revs of the latest changes since seq N from the  
> source.
> Replicator asks the target "do you have these revs?"
> Target responds, "here are the revs I am missing"
> Replicator asks source, give me these revs I am missing or latest  
> revisions of those revs.
> Source returns the docs and attachment info.
> Replicator writes documents and attachments to target
>
>
> For incremental replication, CouchDB would now track which revision  
> a document was edited in, storing the revision number along with the  
> attachment metadata. When an attachment is updated, the revision  
> number is updated along with it. And since we hash the attachments  
> contents, we can be smart an updated it's rev number it only when it  
> actually changes.
>
> New Replication:
>
> Replicator get the revs of the latest changes since seq N from the  
> source.
> Replicator asks the target "do you have these revs?"
> Target responds, "here are the revs I am missing, and these are the  
> latest revs I do have"
> Replicator asks source, give me these revs I am missing or latest  
> revisions of those revs.
> Source returns the docs and attachment info.
> Replicator figures which, if any, earlier revisions of the doc  
> already exist on the target, using the "latest revs" the target gave  
> us.
> Replicator writes to target the documents and only the attachments  
> that have changed since the latest revisions already on target.
>
> These would require some new apis in addition to changes in the  
> replicator, but we can keep around the old apis so old versions can  
> still replicate to us.
>
> Feedback please.
>
> -Damien