You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Mage <ma...@mage.hu> on 2011/04/10 14:59:56 UTC
replication
Hello,
I am new to CouchDB and actually reading the fine Relax book. However
before I spend days or weeks reading about other features I would like
to know that my final goal is accomplishable with CouchDB. This is why I
subscribed.
Suppose that I develop a web app (with Rails) and I setup three CouchDB
nodes: A, B, C. I want all the three to be masters and I want load
balancing (HAProxy?).
As far as I understand I have to setup 3 * 2 replication ways (push and
pull from A to B, A to C and b to C). I need realtime sync (continous
replication?).
The real goal is that if node C goes down, for example because of
network error, then it comes up I'd like to be sure it starts synching
again in all ways (like it did before the error) without manual
interference. It would be also good to be able to restart node C (or B
or A) and have it working again without doing anything on the other
nodes (A and B). I prefer configuration over crontab jobs.
Getting e-mail about conflicts would be also vital for a web app.
Are these things possible with CouchDB or I just have to think a
different way?
I promise I will read the docs soon.
Thank you for any answers.
Mage
Re: replication
Posted by Nebu Pookins <ne...@gmail.com>.
On Sun, Apr 10, 2011 at 8:59 AM, Mage <ma...@mage.hu> wrote:
> I am new to CouchDB and actually reading the fine Relax book.
I'm new to CouchDB too.
> As far as I understand I have to setup 3 * 2 replication ways (push and
> pull from A to B, A to C and b to C). I need realtime sync (continous
> replication?).
>From what I understand, there's a replication mode in CouchDB such
that the CouchDB can keep the connection open, an as changes occur on
the server, information about these changes are set over the
connection so that the client is immediately notified. That's probably
as close to realtime as you can get.
>
> The real goal is that if node C goes down, for example because of
> network error, then it comes up I'd like to be sure it starts synching
> again in all ways (like it did before the error) without manual
> interference.
>
> It would be also good to be able to restart node C (or B
> or A) and have it working again without doing anything on the other
> nodes (A and B). I prefer configuration over crontab jobs.
>From what I heard, CouchDB "forgets" all replicaiton settings on
restart, so you'd need to do this via an external script that can
detect when CouchDB has crashed, and restart it, along with
reconfiguring all of its replications.
>
> Getting e-mail about conflicts would be also vital for a web app.
I haven't seen anything like this in CouchDB, but then again, I'm still a noob.
- Nebu
Re: replication
Posted by Mage <ma...@mage.hu>.
On 04/11/2011 08:40 PM, Nebu Pookins wrote:
> Are you sure the data is actually lost, as opposed to just not visible
> unless you specifically query for the conflicting versions? CouchDB
> has a deterministic "automatic winner" selection algorithm which it
> uses while it waits for your application to select a true winner. See
> http://guide.couchdb.org/draft/conflicts.html
You are right. I haven't read the whole documentation yet. The conflict
is there.
What confused me is that "standard" futon browsing doesn't show the
_conflict attributes. Neither couchrest_model does.
I can see them with a view as mentioned in the conflict chapter you linked.
I plan developing on CouchDB in 6 months and maybe that time
couchrest_model will have attributes for these conflicting revisions. Or
having only one master (write) node can be the solution for
replication-conflicts.
couchrest_model handles the other type of conflicts which happen on
parallel save of the same object. It throws an exception. That's fine
for Rails.
Now I go back reading docs, thank you.
Mage
Re: replication
Posted by Nebu Pookins <ne...@gmail.com>.
On Mon, Apr 11, 2011 at 2:15 PM, Mage <ma...@mage.hu> wrote:
> I mean that
> one version of the document just totally lost when I started the
> replication.
>
> Steps:
>
> Node A Node B
> Version 1 Version 1
> Version 2-A Version 1
> Version 2-A Version 2-B
> ...replication...
> Version 2-B Version 2-B
Are you sure the data is actually lost, as opposed to just not visible
unless you specifically query for the conflicting versions? CouchDB
has a deterministic "automatic winner" selection algorithm which it
uses while it waits for your application to select a true winner. See
http://guide.couchdb.org/draft/conflicts.html
- Nebu
Re: replication
Posted by Mage <ma...@mage.hu>.
Conflicts will occur any time that edits diverge. It's possible for
> conflicts to occur on replication:
>
> * Nodes B & C pull document 1 from node A
> * Nodes B & C make & save separate changes to document 1
> * Replication occurs, either between B & C, or from one node to A and
> then to the other.
>
> Note that continuous replication reduces the time window for conflicts
> to occur, but it will not eliminate them. Thus you must treat conflicts
> not as an "exceptional case" but as an expected state.
I could not trigger conflict straight by the replication. I mean that
one version of the document just totally lost when I started the
replication.
Steps:
Node A Node B
Version 1 Version 1
Version 2-A Version 1
Version 2-A Version 2-B
...replication...
Version 2-B Version 2-B
If I, for example, open two browser windows for the same document on
same database, conflict can happen: I start editing in the first window,
I start editing in the second, I save the second then I save the first.
Conflict happens. This is fair and clear situation.
However if I do the same with two nodes (databases) and I start
replication after the modifying I just lose one version of the document.
On both nodes.
Mage
Re: replication
Posted by Owen Marshall <om...@facilityone.com>.
On 04/11/2011 01:19 PM, Mage wrote:
> On 04/11/2011 06:37 PM, Owen Marshall wrote:
>>
>> IIRC network errors won't kill replication -- it will merely keep trying
>> until the connection is re-established and then come back into sync. But
>> I could be wrong here...
> I pulled out the ethernet cable from the laptop for about 5 minutes then
> plugged back and continous replication didn't get restored.
>
> However calling some cronjob once per minute seems to be acceptable.
> Until a new vesion comes out with permanent replication feature.
I guess I was wrong. Strange -- I was pretty sure that I was right. I
haven't poked the replication in a while, but it is possible that
retries back off when replication fails. Hopefully someone more
knowledgeable can expand on this.
>> Are you *sure*? Think very carefully here.
>>
>> IMO, conflicts should not be seen as "bad". Instead, they should be seen
>> as an intermediate state that should be resolved as soon as possible.
>>
>> If you design your app to watch for _conflicts: true, and handle the
>> conflicts in a way that makes sense to your application, you will be
>> much happier -- not only will your application "just work" without your
>> pager vibrating 24x7, but you may find that your app will be able to do
>> things you hadn't thought of before.
> In the meantime I played a bit with CouchDB and realized that conflicts
> happen when: I start working with a document on node A => I modify and
> save the same document on node B => Replication happens => I try to save
> the original object on node A.
>
> So conflict can happen not during the replication but on save if I am right.
Conflicts will occur any time that edits diverge. It's possible for
conflicts to occur on replication:
* Nodes B & C pull document 1 from node A
* Nodes B & C make & save separate changes to document 1
* Replication occurs, either between B & C, or from one node to A and
then to the other.
Note that continuous replication reduces the time window for conflicts
to occur, but it will not eliminate them. Thus you must treat conflicts
not as an "exceptional case" but as an expected state.
> The strange thing is, at least for me, that if I modify a document on
> node A *when there is no replication* for example if the link is dead
> then I modify the same document on node B and after that I start
> replication then the later version of the document wins and the other
> version becomes "lost" without any trace. It doesn't even stays there as
> previous version. At least as I experienced so far. I might missed
> something.
A basic `GET /db/document` will not provide any indication that a
conflict has occurred. It's up to you to ask.
See here for more:
http://wiki.apache.org/couchdb/Replication_and_conflicts
--
Owen Marshall
FacilityONE
omarshall@facilityone.com | (502) 805-2126
Re: replication
Posted by Nebu Pookins <ne...@gmail.com>.
On Mon, Apr 11, 2011 at 1:19 PM, Mage <ma...@mage.hu> wrote:
> In the meantime I played a bit with CouchDB and realized that conflicts
> happen when: I start working with a document on node A => I modify and
> save the same document on node B => Replication happens => I try to save
> the original object on node A.
>
> So conflict can happen not during the replication but on save if I am right.
I might be mistaken on the terminology, but I believe that this is
distinct from a conflict. Here, CouchDB rejects your update outright,
because when you try to save it, it knows that the version from which
you had based your update was out of date. In contrast, a conflict
occurs long after the user is has finished their update and has left,
but CouchDB later discovers (via replication) that someone else has
also edited, and now no humans are around to tell it what to do.
- Nebu
Re: replication
Posted by Mage <ma...@mage.hu>.
Dear Owen,
On 04/11/2011 06:37 PM, Owen Marshall wrote:
>
> IIRC network errors won't kill replication -- it will merely keep trying
> until the connection is re-established and then come back into sync. But
> I could be wrong here...
I pulled out the ethernet cable from the laptop for about 5 minutes then
plugged back and continous replication didn't get restored.
However calling some cronjob once per minute seems to be acceptable.
Until a new vesion comes out with permanent replication feature.
>
> Are you *sure*? Think very carefully here.
>
> IMO, conflicts should not be seen as "bad". Instead, they should be seen
> as an intermediate state that should be resolved as soon as possible.
>
> If you design your app to watch for _conflicts: true, and handle the
> conflicts in a way that makes sense to your application, you will be
> much happier -- not only will your application "just work" without your
> pager vibrating 24x7, but you may find that your app will be able to do
> things you hadn't thought of before.
In the meantime I played a bit with CouchDB and realized that conflicts
happen when: I start working with a document on node A => I modify and
save the same document on node B => Replication happens => I try to save
the original object on node A.
So conflict can happen not during the replication but on save if I am right.
The strange thing is, at least for me, that if I modify a document on
node A *when there is no replication* for example if the link is dead
then I modify the same document on node B and after that I start
replication then the later version of the document wins and the other
version becomes "lost" without any trace. It doesn't even stays there as
previous version. At least as I experienced so far. I might missed
something.
I am not saying this is wrong I just never used a distributed database
before.
Mage
Re: replication
Posted by Owen Marshall <om...@facilityone.com>.
On 04/10/2011 08:59 AM, Mage wrote:
> As far as I understand I have to setup 3 * 2 replication ways (push and
> pull from A to B, A to C and b to C). I need realtime sync (continous
> replication?).
Correct.
> The real goal is that if node C goes down, for example because of
> network error,
IIRC network errors won't kill replication -- it will merely keep trying
until the connection is re-established and then come back into sync. But
I could be wrong here...
> then it comes up I'd like to be sure it starts synching
> again in all ways (like it did before the error) without manual
> interference.
This is targeted for release in version 1.2, I believe.
Unfortunately for now CouchDB will not "remember" your replication; you
will instead need some out of band process to get everything set the way
you want.
> It would be also good to be able to restart node C (or B
> or A) and have it working again without doing anything on the other
> nodes (A and B). I prefer configuration over crontab jobs
If I was correct above, you should be able to restart node C and merely
re-establish replication on it -- nodes B & A would eventually try to
replicate and succeed.
> Getting e-mail about conflicts would be also vital for a web app.
Are you *sure*? Think very carefully here.
IMO, conflicts should not be seen as "bad". Instead, they should be seen
as an intermediate state that should be resolved as soon as possible.
If you design your app to watch for _conflicts: true, and handle the
conflicts in a way that makes sense to your application, you will be
much happier -- not only will your application "just work" without your
pager vibrating 24x7, but you may find that your app will be able to do
things you hadn't thought of before.
--
Owen Marshall
FacilityONE
omarshall@facilityone.com | (502) 805-2126