progress update on OTP replicator

[Adam Kocoloski <ko...@apache.org>]

Hi again, I put the new replicator through its paces over the past few  
days and am pretty happy with it.  Summary of the changes:

http://github.com/kocolosk/couchdb/commits/otpify-replication

* Only one copy of a given source->target replication can run at any  
given time.  Other POSTs to _replicate with the same request will be  
added as listeners to the ongoing replication.  When that replication  
finishes, we respond to the original requester and check for new  
updates to include in the response to the others.  This is Option 3 in  
the previous thread on dev@

* Replications that terminate with an abnormal reason (other than  
shutdown) will be restarted by the supervisor, but the listeners on  
the original request are just given an {error, reason} response.  They  
can track the results of the retry by re-POSTing.

* Attachments are streamed directly to disk in the case of a pull  
replication.  This feature was made possible by some bleeding-edge  
updates to ibrowse courtesy of Chandru Mullaparthi.  Is still requires  
quite a lot of memory in the ibrowse processes, but we're working on  
that.  The memory footprint in the CouchDB processes is negligible.   
Testing between two EC2 nodes indicated that pulling 400MB of  
attachments was approximately 20x faster than pushing them (push still  
inlines in the JSON).

* Memory utilization has decreased.  All the tests posted by Jeff  
Hinrichs in COUCHDB-270 pass, although the tests with 20MB JSON  
documents still use over a gig of memory in ibrowse processes.

* The parallel asynchronous GET requests we used in pull replications  
have been temporarily rolled back out.  I have a branch that includes  
them, but I want to make sure there are no regressions in the memory  
utilization before merging that feature back in.  As a result, you'll  
find that push replications will be significantly faster than pulls if  
there are no large attachments hanging around.

* Replication updates show up in the Futon status window.

Side note: I redefined couch_util:should_flush() so that it considers  
both the process' memory usage as well as any associated reference- 
counted binaries in calculating its response.  The code to find those  
binaries relies on a part of process_info() that "may be changed or  
removed without prior notice" and isn't terribly well documented, but  
we ignore binary memory usage at our peril.  With the old definition  
should_flush() would come back false even when the replicator had  
grabbed 2GB worth of binaries off the disk!

If there are no objections I'd like to add this code to trunk later  
today.  Best,

Adam

Re: progress update on OTP replicator

[Paul Davis <pa...@gmail.com>]

On Sat, Mar 7, 2009 at 10:18 AM, Adam Kocoloski <ko...@apache.org> wrote:
> Hi again, I put the new replicator through its paces over the past few days
> and am pretty happy with it.  Summary of the changes:
>
> http://github.com/kocolosk/couchdb/commits/otpify-replication
>
> * Only one copy of a given source->target replication can run at any given
> time.  Other POSTs to _replicate with the same request will be added as
> listeners to the ongoing replication.  When that replication finishes, we
> respond to the original requester and check for new updates to include in
> the response to the others.  This is Option 3 in the previous thread on dev@
>
> * Replications that terminate with an abnormal reason (other than shutdown)
> will be restarted by the supervisor, but the listeners on the original
> request are just given an {error, reason} response.  They can track the
> results of the retry by re-POSTing.
>
> * Attachments are streamed directly to disk in the case of a pull
> replication.  This feature was made possible by some bleeding-edge updates
> to ibrowse courtesy of Chandru Mullaparthi.  Is still requires quite a lot
> of memory in the ibrowse processes, but we're working on that.  The memory
> footprint in the CouchDB processes is negligible.  Testing between two EC2
> nodes indicated that pulling 400MB of attachments was approximately 20x
> faster than pushing them (push still inlines in the JSON).
>
> * Memory utilization has decreased.  All the tests posted by Jeff Hinrichs
> in COUCHDB-270 pass, although the tests with 20MB JSON documents still use
> over a gig of memory in ibrowse processes.
>
> * The parallel asynchronous GET requests we used in pull replications have
> been temporarily rolled back out.  I have a branch that includes them, but I
> want to make sure there are no regressions in the memory utilization before
> merging that feature back in.  As a result, you'll find that push
> replications will be significantly faster than pulls if there are no large
> attachments hanging around.
>
> * Replication updates show up in the Futon status window.
>
> Side note: I redefined couch_util:should_flush() so that it considers both
> the process' memory usage as well as any associated reference-counted
> binaries in calculating its response.  The code to find those binaries
> relies on a part of process_info() that "may be changed or removed without
> prior notice" and isn't terribly well documented, but we ignore binary
> memory usage at our peril.  With the old definition should_flush() would
> come back false even when the replicator had grabbed 2GB worth of binaries
> off the disk!
>
> If there are no objections I'd like to add this code to trunk later today.
>  Best,
>
> Adam
>
>

+∞

Re: progress update on OTP replicator

[Chris Anderson <jc...@apache.org>]

On Sat, Mar 7, 2009 at 7:18 AM, Adam Kocoloski <ko...@apache.org> wrote:

> If there are no objections I'd like to add this code to trunk later today.

+1



-- 
Chris Anderson
http://jchris.mfdz.com