You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Damien Katz <da...@apache.org> on 2009/07/15 00:06:35 UTC
Attachment level replication
I've just finished the deterministic revs work, and I've made changes
to the attachment meta data is stored. I figure, since I'm in there
changes things, I should go ahead and make a change to allow for
incremental attachment replication,so we only replicate the binary
attachments (which can be very large) that have changed, instead of
all the attachments any something in the document changes.
Currently when we replicate a changed document, we replicate all the
attachments, regardless if they've changed.
It looks something like this:
Replicator get the revs of the latest changes since seq N from the
source.
Replicator asks the target "do you have these revs?"
Target responds, "here are the revs I am missing"
Replicator asks source, give me these revs I am missing or latest
revisions of those revs.
Source returns the docs and attachment info.
Replicator writes documents and attachments to target
For incremental replication, CouchDB would now track which revision a
document was edited in, storing the revision number along with the
attachment metadata. When an attachment is updated, the revision
number is updated along with it. And since we hash the attachments
contents, we can be smart an updated it's rev number it only when it
actually changes.
New Replication:
Replicator get the revs of the latest changes since seq N from the
source.
Replicator asks the target "do you have these revs?"
Target responds, "here are the revs I am missing, and these are the
latest revs I do have"
Replicator asks source, give me these revs I am missing or latest
revisions of those revs.
Source returns the docs and attachment info.
Replicator figures which, if any, earlier revisions of the doc already
exist on the target, using the "latest revs" the target gave us.
Replicator writes to target the documents and only the attachments
that have changed since the latest revisions already on target.
These would require some new apis in addition to changes in the
replicator, but we can keep around the old apis so old versions can
still replicate to us.
Feedback please.
-Damien
Re: Attachment level replication
Posted by Antony Blakey <an...@gmail.com>.
Excellent. Additionally, if the source and destination share an old
version of the attachment, an xdelta/rsync diff would potentially save
a lot of bandwidth.
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
The intuitive mind is a sacred gift and the rational mind is a
faithful servant. We have created a society that honours the servant
and has forgotten the gift.
-- Albert Einstein
Re: Attachment level replication
Posted by Ben Browning <be...@gmail.com>.
>From what you describe, it sounds like an excellent idea.
Do you have a clear idea of what the new APIs would be and if they're
potentially useful outside of the new replicator?
Also, I don't think it was explicitly mentioned, but is this slated to
make it into a 0.10.x release?
Ben
Re: Attachment level replication
Posted by Adam Kocoloski <ko...@apache.org>.
On Jul 14, 2009, at 6:06 PM, Damien Katz wrote:
> I've just finished the deterministic revs work, and I've made
> changes to the attachment meta data is stored. I figure, since I'm
> in there changes things, I should go ahead and make a change to
> allow for incremental attachment replication,so we only replicate
> the binary attachments (which can be very large) that have changed,
> instead of all the attachments any something in the document changes.
>
> Currently when we replicate a changed document, we replicate all the
> attachments, regardless if they've changed.
>
> It looks something like this:
>
> Replicator get the revs of the latest changes since seq N from the
> source.
> Replicator asks the target "do you have these revs?"
> Target responds, "here are the revs I am missing"
> Replicator asks source, give me these revs I am missing or latest
> revisions of those revs.
> Source returns the docs and attachment info.
> Replicator writes documents and attachments to target
>
>
> For incremental replication, CouchDB would now track which revision
> a document was edited in, storing the revision number along with the
> attachment metadata. When an attachment is updated, the revision
> number is updated along with it. And since we hash the attachments
> contents, we can be smart an updated it's rev number it only when it
> actually changes.
>
> New Replication:
>
> Replicator get the revs of the latest changes since seq N from the
> source.
> Replicator asks the target "do you have these revs?"
> Target responds, "here are the revs I am missing, and these are the
> latest revs I do have"
> Replicator asks source, give me these revs I am missing or latest
> revisions of those revs.
> Source returns the docs and attachment info.
> Replicator figures which, if any, earlier revisions of the doc
> already exist on the target, using the "latest revs" the target gave
> us.
> Replicator writes to target the documents and only the attachments
> that have changed since the latest revisions already on target.
>
> These would require some new apis in addition to changes in the
> replicator, but we can keep around the old apis so old versions can
> still replicate to us.
>
> Feedback please.
>
> -Damien
Sounds good. I think we shouldn't have any trouble preserving
bidirectional replication with 0.9.x, either. Best,
Adam
Re: Attachment level replication
Posted by Zachary Zolton <za...@gmail.com>.
Will 0.10 be replication compatible with 0.9?
Perhaps we could keep a compatibility matrix in the wiki...
On Jul 14, 2009, at 5:06 PM, Damien Katz <da...@apache.org> wrote:
> I've just finished the deterministic revs work, and I've made
> changes to the attachment meta data is stored. I figure, since I'm
> in there changes things, I should go ahead and make a change to
> allow for incremental attachment replication,so we only replicate
> the binary attachments (which can be very large) that have changed,
> instead of all the attachments any something in the document changes.
>
> Currently when we replicate a changed document, we replicate all the
> attachments, regardless if they've changed.
>
> It looks something like this:
>
> Replicator get the revs of the latest changes since seq N from the
> source.
> Replicator asks the target "do you have these revs?"
> Target responds, "here are the revs I am missing"
> Replicator asks source, give me these revs I am missing or latest
> revisions of those revs.
> Source returns the docs and attachment info.
> Replicator writes documents and attachments to target
>
>
> For incremental replication, CouchDB would now track which revision
> a document was edited in, storing the revision number along with the
> attachment metadata. When an attachment is updated, the revision
> number is updated along with it. And since we hash the attachments
> contents, we can be smart an updated it's rev number it only when it
> actually changes.
>
> New Replication:
>
> Replicator get the revs of the latest changes since seq N from the
> source.
> Replicator asks the target "do you have these revs?"
> Target responds, "here are the revs I am missing, and these are the
> latest revs I do have"
> Replicator asks source, give me these revs I am missing or latest
> revisions of those revs.
> Source returns the docs and attachment info.
> Replicator figures which, if any, earlier revisions of the doc
> already exist on the target, using the "latest revs" the target gave
> us.
> Replicator writes to target the documents and only the attachments
> that have changed since the latest revisions already on target.
>
> These would require some new apis in addition to changes in the
> replicator, but we can keep around the old apis so old versions can
> still replicate to us.
>
> Feedback please.
>
> -Damien