replication slow down

[Jean-Yves Moulin <jy...@baaz.fr>]

Hi CouchDB users,

I'm trying to replicate a big database: 35 millions documents. Average doc size is 500 bytes.

When replication starts, from server A to server B (from server B in pull mode), we got replicate at 1100 documents per second. Then, after some thousands of documents replicated, replication slowly decelerates until 100 doc/sec.

If I restart the couchdb daemon on server B, replication restarts at 1100 doc/sec. And then slow down again. At 100 doc/sec I need multiple weeks to replicate the full database.. :-(

Obviously, the two servers are far from their performances limits: one half CPU used (on 16), 10GB of RAM used (on 64GB), and very small I/O (on 24 disks + 2 ssd for cache).

You can see here: http://jym.eileo.net/couchdb_req.png , A small graph showing the http requests on server A (almost all GET requests come from server B). The restart of server B is clearly visible at 10:30.

I already try to change worker_batch_size, worker_processes and http_connections but this doesn't change anything.

Do you know if this is a know bug or how can I improve the situation ?


Thank you very much.

best,
jym

Re: replication slow down

[Jan Lehnardt <ja...@apache.org>]

On Aug 20, 2013, at 18:25 , Jean-Yves Moulin <jy...@baaz.fr> wrote:

> Hi,
> 
> 
> On 20 Aug 2013, at 15:16 , Adam Kocoloski <ko...@apache.org> wrote:
> 
>> Did you specify "continuous": true in the replication spec?  If you did, do you see the same slowdown if you omit that flag?
> 
> Yes, I was using "continuous": true.
> 
> You're right: When disabling "continuous", replication seems more stable and efficient (1200 doc/sec). Thank you !
> 
> Is there a PR open on this, or should I open one ?

Joan opened https://issues.apache.org/jira/browse/COUCHDB-1874 in the meantime. Thanks for the report! :)

Best
Jan
--

Re: replication slow down

[Jean-Yves Moulin <jy...@baaz.fr>]

Hi,


On 20 Aug 2013, at 15:16 , Adam Kocoloski <ko...@apache.org> wrote:

> Did you specify "continuous": true in the replication spec?  If you did, do you see the same slowdown if you omit that flag?

Yes, I was using "continuous": true.

You're right: When disabling "continuous", replication seems more stable and efficient (1200 doc/sec). Thank you !

Is there a PR open on this, or should I open one ?


best,
jym

Re: replication slow down

[Adam Kocoloski <ko...@apache.org>]

Did you specify "continuous": true in the replication spec?  If you did, do you see the same slowdown if you omit that flag?

This sounds rather similar to a bug we encountered in early builds of BigCouch, but I don't remember the history of the codebase well enough to say whether the bug might be latent in Apache CouchDB.

Adam

On Aug 20, 2013, at 8:49 AM, Jean-Yves Moulin <jy...@baaz.fr> wrote:

> Hi CouchDB users,
> 
> I'm trying to replicate a big database: 35 millions documents. Average doc size is 500 bytes.
> 
> When replication starts, from server A to server B (from server B in pull mode), we got replicate at 1100 documents per second. Then, after some thousands of documents replicated, replication slowly decelerates until 100 doc/sec.
> 
> If I restart the couchdb daemon on server B, replication restarts at 1100 doc/sec. And then slow down again. At 100 doc/sec I need multiple weeks to replicate the full database.. :-(
> 
> Obviously, the two servers are far from their performances limits: one half CPU used (on 16), 10GB of RAM used (on 64GB), and very small I/O (on 24 disks + 2 ssd for cache).
> 
> You can see here: http://jym.eileo.net/couchdb_req.png , A small graph showing the http requests on server A (almost all GET requests come from server B). The restart of server B is clearly visible at 10:30.
> 
> I already try to change worker_batch_size, worker_processes and http_connections but this doesn't change anything.
> 
> Do you know if this is a know bug or how can I improve the situation ?
> 
> 
> Thank you very much.
> 
> best,
> jym