You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Benoit Chesneau <bc...@gmail.com> on 2009/09/20 19:06:49 UTC

continuous replication questions

Hi,

I've done today some tests with replications (currently 1000 continous
replications open). From a remote node to another. So I have 1000 http
connections opened from one node to another. Seem to work well.

But I wonder what happened if one replication link hangs. Is there a
way to know a database isn't anymore replicated except by parsing
status ? Could we retrieve the lists of running replications maybe not
by HTTP but with Erlang ? Would be useful since such thing could
happened on so many open connections.

About that I think we need a way to replicate in real time a full
couchdb node to another in one http connection, it will cost less
memory (actually 193M are used for 1000 connections) and less open
files. I don't know if it's possible by just using the current update
events ? Or maybe by doing something like stats by adding a function ?
For global replication I think that only having db informations in the
feed is enough, so we get the last updated seq and if the db changed.
Then replicator, could get the list of changes since the last updated
seq it stored. What do you think about it ?

- benoît

Re: continuous replication questions

Posted by Adam Kocoloski <ko...@apache.org>.
On Sep 20, 2009, at 1:39 PM, Benoit Chesneau wrote:

> On suggestion of jan I repost it here.
>
> ---------- Forwarded message ----------
> From: Benoit Chesneau <bc...@gmail.com>
> Date: Sun, Sep 20, 2009 at 7:06 PM
> Subject: continuous replication questions
> To: user@couchdb.apache.org
>
>
> Hi,
>
> I've done today some tests with replications (currently 1000 continous
> replications open). From a remote node to another. So I have 1000 http
> connections opened from one node to another. Seem to work well.
>
> But I wonder what happened if one replication link hangs. Is there a
> way to know a database isn't anymore replicated except by parsing
> status ? Could we retrieve the lists of running replications maybe not
> by HTTP but with Erlang ? Would be useful since such thing could
> happened on so many open connections.

In Erlang you can do

supervisor:which_children(couch_rep_sup)

to get the full list of replications initiated on that node since the  
last VM restart.  Any replication with an undefined process id is no  
longer running.  It's of limited utility, though, because the IDs are  
opaque and there's no easy way to figure out which IDs map to which  
replication configurations.  _active_tasks remains your best bet for  
now.

Regards, Adam



Re: continuous replication questions

Posted by Benoit Chesneau <bc...@gmail.com>.
On Mon, Sep 21, 2009 at 9:32 AM, Benoit Chesneau <bc...@gmail.com> wrote:
> This morning I wrote a quick patch for a global db changes handler :
>
> https://issues.apache.org/jira/secure/attachment/12420170/db_changes.patch
>
> It manage longpolling and continuous (set feed=continuous, default is
> longpoll).
>
> Each change send a line
> {"type": "typeofchange", "db": "nameof db", "seq": "updated seq in db"}
>
> Timeout and heartbeat can also be set. Could be used in place of
> current update db notifier. This patch is in dbchanges branch of my
> repo :
>
> http://bitbucket.org/benoitc/couchdb/src/b8373046523b/
>
>
> - benoît
>

patch and repo have been updated with tests :

https://issues.apache.org/jira/secure/attachment/12420174/db_changes2.patch

- benoit

Re: continuous replication questions

Posted by Benoit Chesneau <bc...@gmail.com>.
On Sun, Sep 20, 2009 at 7:39 PM, Benoit Chesneau <bc...@gmail.com> wrote:

>
> About that I think we need a way to replicate in real time a full
> couchdb node to another in one http connection, it will cost less
> memory (actually 193M are used for 1000 connections) and less open
> files. I don't know if it's possible by just using the current update
> events ? Or maybe by doing something like stats by adding a function ?
> For global replication I think that only having db informations in the
> feed is enough, so we get the last updated seq and if the db changed.
> Then replicator, could get the list of changes since the last updated
> seq it stored. What do you think about it ?
>

This morning I wrote a quick patch for a global db changes handler :

https://issues.apache.org/jira/secure/attachment/12420170/db_changes.patch

It manage longpolling and continuous (set feed=continuous, default is
longpoll).

Each change send a line
{"type": "typeofchange", "db": "nameof db", "seq": "updated seq in db"}

Timeout and heartbeat can also be set. Could be used in place of
current update db notifier. This patch is in dbchanges branch of my
repo :

http://bitbucket.org/benoitc/couchdb/src/b8373046523b/


- benoît

continuous replication questions

Posted by Benoit Chesneau <bc...@gmail.com>.
On suggestion of jan I repost it here.

---------- Forwarded message ----------
From: Benoit Chesneau <bc...@gmail.com>
Date: Sun, Sep 20, 2009 at 7:06 PM
Subject: continuous replication questions
To: user@couchdb.apache.org


Hi,

I've done today some tests with replications (currently 1000 continous
replications open). From a remote node to another. So I have 1000 http
connections opened from one node to another. Seem to work well.

But I wonder what happened if one replication link hangs. Is there a
way to know a database isn't anymore replicated except by parsing
status ? Could we retrieve the lists of running replications maybe not
by HTTP but with Erlang ? Would be useful since such thing could
happened on so many open connections.

About that I think we need a way to replicate in real time a full
couchdb node to another in one http connection, it will cost less
memory (actually 193M are used for 1000 connections) and less open
files. I don't know if it's possible by just using the current update
events ? Or maybe by doing something like stats by adding a function ?
For global replication I think that only having db informations in the
feed is enough, so we get the last updated seq and if the db changed.
Then replicator, could get the list of changes since the last updated
seq it stored. What do you think about it ?

- benoît

Re: continuous replication questions

Posted by Benoit Chesneau <bc...@gmail.com>.
On Sun, Sep 20, 2009 at 9:55 PM, Matt Aimonetti <ma...@gmail.com> wrote:
> Benoit, just out of curiosity, can you explain how you setup the 1000
> continuous replications?
>
> Thanks,
>
> - Matt

I've 2 amchines here for the tests. When I decide to create one db,
it's created in the same time on 2 machine using couchdbkit, once dbs
are created I setup replication on second node :

POST /_replicate {"source": "http://node0/somedb", target="somedb",
"continuous": "true"} and that's it :)

So now each time I create a doc on node0 it's automatically replicated on node1.

- benoit

Re: continuous replication questions

Posted by Matt Aimonetti <ma...@gmail.com>.
Benoit, just out of curiosity, can you explain how you setup the 1000
continuous replications?

Thanks,

- Matt

On Sun, Sep 20, 2009 at 10:06 AM, Benoit Chesneau <bc...@gmail.com>wrote:

> Hi,
>
> I've done today some tests with replications (currently 1000 continous
> replications open). From a remote node to another. So I have 1000 http
> connections opened from one node to another. Seem to work well.
>
> But I wonder what happened if one replication link hangs. Is there a
> way to know a database isn't anymore replicated except by parsing
> status ? Could we retrieve the lists of running replications maybe not
> by HTTP but with Erlang ? Would be useful since such thing could
> happened on so many open connections.
>
> About that I think we need a way to replicate in real time a full
> couchdb node to another in one http connection, it will cost less
> memory (actually 193M are used for 1000 connections) and less open
> files. I don't know if it's possible by just using the current update
> events ? Or maybe by doing something like stats by adding a function ?
> For global replication I think that only having db informations in the
> feed is enough, so we get the last updated seq and if the db changed.
> Then replicator, could get the list of changes since the last updated
> seq it stored. What do you think about it ?
>
> - benoît
>