You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Nicholas Orr <ni...@zxgen.net> on 2009/10/01 03:27:22 UTC
Re: replication feedback (was Re: [VOTE] Apache CouchDB 0.10.0
release)
Thanks Adam,
Hmm status page huh? let me check that out :)
Starting things up again and having a look....
Ok the status page does show stuff happening and then it stopped
updating, and just shows this
Replication 70c214: http://192.168.1.11:5984/many_docs/ ->
test <0.136.0> W Processed source update #18270
Is that suppose to go away as all the docs have been replicated.
Nick
On Thu, Oct 1, 2009 at 12:24 AM, Adam Kocoloski <ko...@apache.org> wrote:
> On Sep 30, 2009, at 3:14 AM, Nicholas Orr wrote:
>
>> On Wed, Sep 30, 2009 at 4:24 PM, Nicholas Orr <ni...@zxgen.net>
>> wrote:
>>>
>>> - compiled and run on Gentoo amd64 / erlang R13B01
>>> - futon tests passed
>>> - able to replicate db from 0.9.0, however futon didn't really give
>>> accurate feedback as to what was happening, everything replicated fine
>>> though :/
>>> - couchrest 0.34 doesn't appear to work out of the box yet
>>
>> I got this wrong, was connecting to the wrong database.
>> App works fine.
>>
>> all systems go for me
>> (I don't think the lack of feedback for replication in futon should
>> hold things up..)
>>
>> +1
>
> Hi Nick, did you check the Status page in Futon while the replication was
> running? Updates should've shown up there. I agree with you that it would
> be good to show some feedback on the Replicator page. Best,
>
> Adam
>
Re: replication feedback (was Re: [VOTE] Apache CouchDB 0.10.0
release)
Posted by Nicholas Orr <ni...@zxgen.net>.
Well it hasn't ;)
Unless futon does "continuous:" true - I don't see anything in futon
to specify that option.
Nick
On Thu, Oct 1, 2009 at 11:41 AM, Adam Kocoloski <ko...@apache.org> wrote:
> Yes, that message should have disappeared at the end, unless you did a
> replication with the "continuous":true option. Best,
>
> Adam
>
> On Sep 30, 2009, at 9:27 PM, Nicholas Orr wrote:
>
>> Thanks Adam,
>>
>> Hmm status page huh? let me check that out :)
>> Starting things up again and having a look....
>>
>> Ok the status page does show stuff happening and then it stopped
>> updating, and just shows this
>>
>> Replication 70c214: http://192.168.1.11:5984/many_docs/ ->
>> test <0.136.0> W Processed source update #18270
>>
>> Is that suppose to go away as all the docs have been replicated.
>>
>> Nick
>>
>> On Thu, Oct 1, 2009 at 12:24 AM, Adam Kocoloski <ko...@apache.org>
>> wrote:
>>>
>>> On Sep 30, 2009, at 3:14 AM, Nicholas Orr wrote:
>>>
>>>> On Wed, Sep 30, 2009 at 4:24 PM, Nicholas Orr <ni...@zxgen.net>
>>>> wrote:
>>>>>
>>>>> - compiled and run on Gentoo amd64 / erlang R13B01
>>>>> - futon tests passed
>>>>> - able to replicate db from 0.9.0, however futon didn't really give
>>>>> accurate feedback as to what was happening, everything replicated fine
>>>>> though :/
>>>>> - couchrest 0.34 doesn't appear to work out of the box yet
>>>>
>>>> I got this wrong, was connecting to the wrong database.
>>>> App works fine.
>>>>
>>>> all systems go for me
>>>> (I don't think the lack of feedback for replication in futon should
>>>> hold things up..)
>>>>
>>>> +1
>>>
>>> Hi Nick, did you check the Status page in Futon while the replication was
>>> running? Updates should've shown up there. I agree with you that it
>>> would
>>> be good to show some feedback on the Replicator page. Best,
>>>
>>> Adam
>>>
>
>
Re: replication feedback (was Re: [VOTE] Apache CouchDB 0.10.0 release)
Posted by Adam Kocoloski <ko...@apache.org>.
Yes, that message should have disappeared at the end, unless you did a
replication with the "continuous":true option. Best,
Adam
On Sep 30, 2009, at 9:27 PM, Nicholas Orr wrote:
> Thanks Adam,
>
> Hmm status page huh? let me check that out :)
> Starting things up again and having a look....
>
> Ok the status page does show stuff happening and then it stopped
> updating, and just shows this
>
> Replication 70c214: http://192.168.1.11:5984/many_docs/ ->
> test <0.136.0> W Processed source update #18270
>
> Is that suppose to go away as all the docs have been replicated.
>
> Nick
>
> On Thu, Oct 1, 2009 at 12:24 AM, Adam Kocoloski
> <ko...@apache.org> wrote:
>> On Sep 30, 2009, at 3:14 AM, Nicholas Orr wrote:
>>
>>> On Wed, Sep 30, 2009 at 4:24 PM, Nicholas Orr <nicholas.orr@zxgen.net
>>> >
>>> wrote:
>>>>
>>>> - compiled and run on Gentoo amd64 / erlang R13B01
>>>> - futon tests passed
>>>> - able to replicate db from 0.9.0, however futon didn't really give
>>>> accurate feedback as to what was happening, everything replicated
>>>> fine
>>>> though :/
>>>> - couchrest 0.34 doesn't appear to work out of the box yet
>>>
>>> I got this wrong, was connecting to the wrong database.
>>> App works fine.
>>>
>>> all systems go for me
>>> (I don't think the lack of feedback for replication in futon should
>>> hold things up..)
>>>
>>> +1
>>
>> Hi Nick, did you check the Status page in Futon while the
>> replication was
>> running? Updates should've shown up there. I agree with you that
>> it would
>> be good to show some feedback on the Replicator page. Best,
>>
>> Adam
>>
Re: replication feedback (was Re: [VOTE] Apache CouchDB 0.10.0
release)
Posted by Matt Goodall <ma...@gmail.com>.
2009/10/1 Adam Kocoloski <ko...@apache.org>:
> On Oct 1, 2009, at 8:57 AM, Matt Goodall wrote:
>
>> 2009/10/1 Nicholas Orr <ni...@zxgen.net>:
>>>
>>> Thanks Adam,
>>>
>>> Hmm status page huh? let me check that out :)
>>> Starting things up again and having a look....
>>>
>>> Ok the status page does show stuff happening and then it stopped
>>> updating, and just shows this
>>>
>>> Replication 70c214: http://192.168.1.11:5984/many_docs/ ->
>>> test <0.136.0> W Processed source update #18270
>>>
>>> Is that suppose to go away as all the docs have been replicated.
>>
>> Hi,
>>
>> I'm seeing something similar. The replication process stays in the
>> status list until couchdb is restarted, however my replication is
>> *not* completing. If I restart the controlling couchdb and the
>> replication process it continues from where it left off but often
>> stalls again. Eventually, after a number of couchdb and replication
>> restarts it reaches the end.
>>
>> In case it's useful here's the essence of the log files for a pull
>> replication:
>
> Nope, nothing useful in those logs -- everything looks normal. Is this a
> continuous replication? If so, it seems like it could be related to the
> earlier thread about replication terminating prematurely. Are any proxies
> involved?
Not continuous, I might try that later.
No proxies involved. The two machines were on the same network, not
even a firewall between them in fact.
>
>> <logs snipped>
>
>> I was actually really surprised to see the _local and
>> _ensure_full_commit requests in the source database's logs. Is that
>> correct behaviour?
>
> Yes, it is correct.
>
> The _ensure_full_commit is to make sure that all replicated documents are
> safely committed to disk. If we don't do this, we run the risk of skipping
> documents in replication in the future (if the source restarts and loses
> documents, it will reuse some update sequences, but the replicator will skip
> them because it thinks that it already replicated them).
>
> The _local document is used to record the replication checkpoints and
> history. We record it on both the source and the target so that if the
> source DB is deleted and recreated, replication will start again from 0.
Ah, ok. Thanks for the explanation.
>
>> I'm sure someone reported problems with the replication process dieing
>> and having to restart replication until it finally gets to end but I
>> can't find the email now. To be honest, that would be fine -
>> restarting just replication is easy and non-intrusive - but needing to
>> restart the couchdb server to clear the replication is not nice.
>
> Agreed, will try to find a way to reproduce this. Best,
>
> Adam
>
Re: replication feedback (was Re: [VOTE] Apache CouchDB 0.10.0 release)
Posted by Adam Kocoloski <ko...@apache.org>.
On Oct 1, 2009, at 8:57 AM, Matt Goodall wrote:
> 2009/10/1 Nicholas Orr <ni...@zxgen.net>:
>> Thanks Adam,
>>
>> Hmm status page huh? let me check that out :)
>> Starting things up again and having a look....
>>
>> Ok the status page does show stuff happening and then it stopped
>> updating, and just shows this
>>
>> Replication 70c214: http://192.168.1.11:5984/many_docs/ ->
>> test <0.136.0> W Processed source update #18270
>>
>> Is that suppose to go away as all the docs have been replicated.
>
> Hi,
>
> I'm seeing something similar. The replication process stays in the
> status list until couchdb is restarted, however my replication is
> *not* completing. If I restart the controlling couchdb and the
> replication process it continues from where it left off but often
> stalls again. Eventually, after a number of couchdb and replication
> restarts it reaches the end.
>
> In case it's useful here's the essence of the log files for a pull
> replication:
Nope, nothing useful in those logs -- everything looks normal. Is
this a continuous replication? If so, it seems like it could be
related to the earlier thread about replication terminating
prematurely. Are any proxies involved?
> <logs snipped>
> I was actually really surprised to see the _local and
> _ensure_full_commit requests in the source database's logs. Is that
> correct behaviour?
Yes, it is correct.
The _ensure_full_commit is to make sure that all replicated documents
are safely committed to disk. If we don't do this, we run the risk of
skipping documents in replication in the future (if the source
restarts and loses documents, it will reuse some update sequences, but
the replicator will skip them because it thinks that it already
replicated them).
The _local document is used to record the replication checkpoints and
history. We record it on both the source and the target so that if
the source DB is deleted and recreated, replication will start again
from 0.
> I'm sure someone reported problems with the replication process dieing
> and having to restart replication until it finally gets to end but I
> can't find the email now. To be honest, that would be fine -
> restarting just replication is easy and non-intrusive - but needing to
> restart the couchdb server to clear the replication is not nice.
Agreed, will try to find a way to reproduce this. Best,
Adam
Re: replication feedback (was Re: [VOTE] Apache CouchDB 0.10.0
release)
Posted by Matt Goodall <ma...@gmail.com>.
2009/10/1 Nicholas Orr <ni...@zxgen.net>:
> Thanks Adam,
>
> Hmm status page huh? let me check that out :)
> Starting things up again and having a look....
>
> Ok the status page does show stuff happening and then it stopped
> updating, and just shows this
>
> Replication 70c214: http://192.168.1.11:5984/many_docs/ ->
> test <0.136.0> W Processed source update #18270
>
> Is that suppose to go away as all the docs have been replicated.
Hi,
I'm seeing something similar. The replication process stays in the
status list until couchdb is restarted, however my replication is
*not* completing. If I restart the controlling couchdb and the
replication process it continues from where it left off but often
stalls again. Eventually, after a number of couchdb and replication
restarts it reaches the end.
In case it's useful here's the essence of the log files for a pull replication:
target.log (the couchdb in control)
[info] [<0.115.0>] Replication records differ. Scanning histories to
find a common ancestor.
[info] [<0.115.0>] no common ancestry -- performing full replication
[info] [<0.71.0>] starting new replication
"958b924bcb0fc00e55b442c88943e316" at <0.115.0>
[info] [<0.115.0>] recording a checkpoint at source update_seq 1
[... a number of "recording a checkpoint" lines ..]
[info] [<0.115.0>] recording a checkpoint at source update_seq 4760
source.log
[info] [<0.62.0>] 192.168.0.10 - - 'HEAD' /mydb/ 200
[info] [<0.92.0>] 192.168.0.10 - - 'GET'
/mydb/_local%2F958b924bcb0fc00e55b442c88943e316 200
[info] [<0.99.0>] 192.168.0.10 - - 'GET' /mydb/ 200
[info] [<0.102.0>] 192.168.0.10 - - 'GET'
/mydb/df9b3667e5904610a49a7d49d7d8a025?open_revs=["1-bcece32a577db67c6aae5ce0dfeb1ed7"]&revs=true&latest=true
200
[info] [<0.104.0>] 192.168.0.10 - - 'GET'
/mydb/654b52bae2b74ef09732ee87f6ae75a1?open_revs=["1-479d5f2ea0be7ff4d901e784e9f12d44"]&revs=true&latest=true
200
[... lots of GETs ...]
[info] [<0.111.0>] 192.168.0.10 - - 'POST' /mydb/_ensure_full_commit?seq=1 201
[info] [<0.62.0>] 192.168.0.10 - - 'GET'
/mydb/3631a680e851434cbeee2b3953228ccd?open_revs=["1-e35eb2f531fe9ecf74eccf329a270bdf"]&revs=true&latest=true
200
[... lots of GETs ...]
[info] [<0.113.0>] 192.168.0.10 - - 'PUT'
/mydb/_local%2F958b924bcb0fc00e55b442c88943e316 201
[info] [<0.99.0>] 192.168.0.10 - - 'GET'
/mydb/493f23b7f4d04dd9868fd9024fde7476?open_revs=["1-ed05b008038784417c05a6ea4da1064a"]&revs=true&latest=true
200
[... lots of requests ...]
[info] [<0.62.0>] 192.168.0.10 - - 'GET'
/mydb/ff9b0ff93dbe4482af6175bba8d67a51/ac48e55f38ab4ed5ba61f466454f20a6?rev=1-58ac55058f1df04d0fa43c42814adeaf
200
[info] [<0.301.0>] 192.168.0.10 - - 'POST'
/mydb/_ensure_full_commit?seq=4760 201
[info] [<0.307.0>] 192.168.0.10 - - 'PUT'
/mydb/_local%2F958b924bcb0fc00e55b442c88943e316 201
I was actually really surprised to see the _local and
_ensure_full_commit requests in the source database's logs. Is that
correct behaviour?
I'm sure someone reported problems with the replication process dieing
and having to restart replication until it finally gets to end but I
can't find the email now. To be honest, that would be fine -
restarting just replication is easy and non-intrusive - but needing to
restart the couchdb server to clear the replication is not nice.
- Matt
>
> Nick
>
> On Thu, Oct 1, 2009 at 12:24 AM, Adam Kocoloski <ko...@apache.org> wrote:
>> On Sep 30, 2009, at 3:14 AM, Nicholas Orr wrote:
>>
>>> On Wed, Sep 30, 2009 at 4:24 PM, Nicholas Orr <ni...@zxgen.net>
>>> wrote:
>>>>
>>>> - compiled and run on Gentoo amd64 / erlang R13B01
>>>> - futon tests passed
>>>> - able to replicate db from 0.9.0, however futon didn't really give
>>>> accurate feedback as to what was happening, everything replicated fine
>>>> though :/
>>>> - couchrest 0.34 doesn't appear to work out of the box yet
>>>
>>> I got this wrong, was connecting to the wrong database.
>>> App works fine.
>>>
>>> all systems go for me
>>> (I don't think the lack of feedback for replication in futon should
>>> hold things up..)
>>>
>>> +1
>>
>> Hi Nick, did you check the Status page in Futon while the replication was
>> running? Updates should've shown up there. I agree with you that it would
>> be good to show some feedback on the Replicator page. Best,
>>
>> Adam
>>
>