You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Brad Rhoads <bd...@gmail.com> on 2013/10/23 23:04:44 UTC
Doc Not Restored After PURGE
Say I have two databases, theSource and theTarget, then:
Add docs 1,2 & 3 to theSource.
Replicate theSource to theTarget.
PURGE doc 2 from theTarget.
Replicate theSource to theTarget again.
I'd expect doc 2 to be restored to theTarget, but it is not being resorted.
I've tested this in 1.2 and 1.3. I don't have 1.4 setup yet, so I haven't
tested this senario on 1.4.
Am I misunderstanding how this is supposed to work, or is there a bug?
Thanks!
-Brad
---------------------------
www.maf.org/rhoads
www.ontherhoads.org
Re: Doc Not Restored After PURGE
Posted by Jens Alfke <je...@couchbase.com>.
On Oct 23, 2013, at 3:07 PM, Jay Clark <jc...@maf.org> wrote:
> What about if a COMPACT were performed on theTarget before replicating again from theSource? Does that affect replication and which docs get checked?
No, compact doesn’t affect sequence numbers or replication checkpoints. (In normal circumstances you don’t want checkpoints to be invalidated, because it makes the next replication really slow, as it has to compare the revisions of every single document.)
There might be a property you can set in the replication’s JSON spec to force a reset from sequence 0; I can’t recall now whether that exists or what it’s called.
—Jens
Re: Doc Not Restored After PURGE
Posted by Adam Kocoloski <ad...@gmail.com>.
Nope. The only thing you can do on the target that affects which documents on the source are checked is modify / delete the _local document storing the checkpoint information. The ID of that document is available from the doc specifying the replication in the _replicator DB.
Adam
On Oct 23, 2013, at 6:07 PM, Jay Clark <jc...@maf.org> wrote:
> What about if a COMPACT were performed on theTarget before replicating again from theSource? Does that affect replication and which docs get checked?
>
>
> ~Jay
>
>
>
> -------- Original message --------
> From Adam Kocoloski <ad...@gmail.com>
> Date: 2013/10/23 4:03 PM (GMT-07:00)
> To user@couchdb.apache.org
> Subject Re: Doc Not Restored After PURGE
>
>
> That's correct. What _purge doesn't do is inform the replicator that some of the work it had done is no longer valid.
>
> The replicator tracks the last sequence on the source DB that it has replicated to the target. When you replicate after the purge the replicator has no reason to revisit doc2 (it only cares about the feed of updates on the source), so it doesn't. If you bump the rev of doc2 on the source the replicator re-examines it and transfers it over.
>
> Make sense?
>
> Adam
>
> On Oct 23, 2013, at 5:56 PM, Brad Rhoads <bd...@gmail.com> wrote:
>
>> That's what I would expect from a DELETE, but I thought _purge was supposed
>> to make the database as if the doc was never there?
>>
>> ---------------------------
>> www.maf.org/rhoads<http://www.maf.org/rhoads>
>> www.ontherhoads.org<http://www.ontherhoads.org>
>>
>>
>> On Wed, Oct 23, 2013 at 3:49 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>
>>> That's definitely one of the dangerous bits about purge. If you do a noop
>>> update to doc2 on theSource and then replicate it should make its way to
>>> theTarget.
>>>
>>> Adam
>>>
>>> On Oct 23, 2013, at 5:04 PM, Brad Rhoads <bd...@gmail.com> wrote:
>>>
>>>> Say I have two databases, theSource and theTarget, then:
>>>>
>>>> Add docs 1,2 & 3 to theSource.
>>>> Replicate theSource to theTarget.
>>>> PURGE doc 2 from theTarget.
>>>> Replicate theSource to theTarget again.
>>>>
>>>> I'd expect doc 2 to be restored to theTarget, but it is not being
>>> resorted.
>>>>
>>>> I've tested this in 1.2 and 1.3. I don't have 1.4 setup yet, so I haven't
>>>> tested this senario on 1.4.
>>>>
>>>> Am I misunderstanding how this is supposed to work, or is there a bug?
>>>>
>>>> Thanks!
>>>>
>>>> -Brad
>>>>
>>>>
>>>> ---------------------------
>>>> www.maf.org/rhoads<http://www.maf.org/rhoads>
>>>> www.ontherhoads.org<http://www.ontherhoads.org>
>>>
>>>
Re: Doc Not Restored After PURGE
Posted by Jay Clark <jc...@maf.org>.
What about if a COMPACT were performed on theTarget before replicating again from theSource? Does that affect replication and which docs get checked?
~Jay
-------- Original message --------
>From Adam Kocoloski <ad...@gmail.com>
Date: 2013/10/23 4:03 PM (GMT-07:00)
To user@couchdb.apache.org
Subject Re: Doc Not Restored After PURGE
That's correct. What _purge doesn't do is inform the replicator that some of the work it had done is no longer valid.
The replicator tracks the last sequence on the source DB that it has replicated to the target. When you replicate after the purge the replicator has no reason to revisit doc2 (it only cares about the feed of updates on the source), so it doesn't. If you bump the rev of doc2 on the source the replicator re-examines it and transfers it over.
Make sense?
Adam
On Oct 23, 2013, at 5:56 PM, Brad Rhoads <bd...@gmail.com> wrote:
> That's what I would expect from a DELETE, but I thought _purge was supposed
> to make the database as if the doc was never there?
>
> ---------------------------
> www.maf.org/rhoads<http://www.maf.org/rhoads>
> www.ontherhoads.org<http://www.ontherhoads.org>
>
>
> On Wed, Oct 23, 2013 at 3:49 PM, Adam Kocoloski <ko...@apache.org> wrote:
>
>> That's definitely one of the dangerous bits about purge. If you do a noop
>> update to doc2 on theSource and then replicate it should make its way to
>> theTarget.
>>
>> Adam
>>
>> On Oct 23, 2013, at 5:04 PM, Brad Rhoads <bd...@gmail.com> wrote:
>>
>>> Say I have two databases, theSource and theTarget, then:
>>>
>>> Add docs 1,2 & 3 to theSource.
>>> Replicate theSource to theTarget.
>>> PURGE doc 2 from theTarget.
>>> Replicate theSource to theTarget again.
>>>
>>> I'd expect doc 2 to be restored to theTarget, but it is not being
>> resorted.
>>>
>>> I've tested this in 1.2 and 1.3. I don't have 1.4 setup yet, so I haven't
>>> tested this senario on 1.4.
>>>
>>> Am I misunderstanding how this is supposed to work, or is there a bug?
>>>
>>> Thanks!
>>>
>>> -Brad
>>>
>>>
>>> ---------------------------
>>> www.maf.org/rhoads<http://www.maf.org/rhoads>
>>> www.ontherhoads.org<http://www.ontherhoads.org>
>>
>>
Re: Doc Not Restored After PURGE
Posted by Adam Kocoloski <ad...@gmail.com>.
That's correct. What _purge doesn't do is inform the replicator that some of the work it had done is no longer valid.
The replicator tracks the last sequence on the source DB that it has replicated to the target. When you replicate after the purge the replicator has no reason to revisit doc2 (it only cares about the feed of updates on the source), so it doesn't. If you bump the rev of doc2 on the source the replicator re-examines it and transfers it over.
Make sense?
Adam
On Oct 23, 2013, at 5:56 PM, Brad Rhoads <bd...@gmail.com> wrote:
> That's what I would expect from a DELETE, but I thought _purge was supposed
> to make the database as if the doc was never there?
>
> ---------------------------
> www.maf.org/rhoads
> www.ontherhoads.org
>
>
> On Wed, Oct 23, 2013 at 3:49 PM, Adam Kocoloski <ko...@apache.org> wrote:
>
>> That's definitely one of the dangerous bits about purge. If you do a noop
>> update to doc2 on theSource and then replicate it should make its way to
>> theTarget.
>>
>> Adam
>>
>> On Oct 23, 2013, at 5:04 PM, Brad Rhoads <bd...@gmail.com> wrote:
>>
>>> Say I have two databases, theSource and theTarget, then:
>>>
>>> Add docs 1,2 & 3 to theSource.
>>> Replicate theSource to theTarget.
>>> PURGE doc 2 from theTarget.
>>> Replicate theSource to theTarget again.
>>>
>>> I'd expect doc 2 to be restored to theTarget, but it is not being
>> resorted.
>>>
>>> I've tested this in 1.2 and 1.3. I don't have 1.4 setup yet, so I haven't
>>> tested this senario on 1.4.
>>>
>>> Am I misunderstanding how this is supposed to work, or is there a bug?
>>>
>>> Thanks!
>>>
>>> -Brad
>>>
>>>
>>> ---------------------------
>>> www.maf.org/rhoads
>>> www.ontherhoads.org
>>
>>
Re: Doc Not Restored After PURGE
Posted by Brad Rhoads <bd...@gmail.com>.
That's what I would expect from a DELETE, but I thought _purge was supposed
to make the database as if the doc was never there?
---------------------------
www.maf.org/rhoads
www.ontherhoads.org
On Wed, Oct 23, 2013 at 3:49 PM, Adam Kocoloski <ko...@apache.org> wrote:
> That's definitely one of the dangerous bits about purge. If you do a noop
> update to doc2 on theSource and then replicate it should make its way to
> theTarget.
>
> Adam
>
> On Oct 23, 2013, at 5:04 PM, Brad Rhoads <bd...@gmail.com> wrote:
>
> > Say I have two databases, theSource and theTarget, then:
> >
> > Add docs 1,2 & 3 to theSource.
> > Replicate theSource to theTarget.
> > PURGE doc 2 from theTarget.
> > Replicate theSource to theTarget again.
> >
> > I'd expect doc 2 to be restored to theTarget, but it is not being
> resorted.
> >
> > I've tested this in 1.2 and 1.3. I don't have 1.4 setup yet, so I haven't
> > tested this senario on 1.4.
> >
> > Am I misunderstanding how this is supposed to work, or is there a bug?
> >
> > Thanks!
> >
> > -Brad
> >
> >
> > ---------------------------
> > www.maf.org/rhoads
> > www.ontherhoads.org
>
>
Re: Doc Not Restored After PURGE
Posted by Adam Kocoloski <ko...@apache.org>.
That's definitely one of the dangerous bits about purge. If you do a noop update to doc2 on theSource and then replicate it should make its way to theTarget.
Adam
On Oct 23, 2013, at 5:04 PM, Brad Rhoads <bd...@gmail.com> wrote:
> Say I have two databases, theSource and theTarget, then:
>
> Add docs 1,2 & 3 to theSource.
> Replicate theSource to theTarget.
> PURGE doc 2 from theTarget.
> Replicate theSource to theTarget again.
>
> I'd expect doc 2 to be restored to theTarget, but it is not being resorted.
>
> I've tested this in 1.2 and 1.3. I don't have 1.4 setup yet, so I haven't
> tested this senario on 1.4.
>
> Am I misunderstanding how this is supposed to work, or is there a bug?
>
> Thanks!
>
> -Brad
>
>
> ---------------------------
> www.maf.org/rhoads
> www.ontherhoads.org
Re: Doc Not Restored After PURGE
Posted by Dave Cottlehuber <dc...@jsonified.com>.
On 23. Oktober 2013 at 23:05:47, Brad Rhoads (bdrhoa@gmail.com) wrote:
>
>Say I have two databases, theSource and theTarget, then:
>
>Add docs 1,2 & 3 to theSource.
>Replicate theSource to theTarget.
>PURGE doc 2 from theTarget.
>Replicate theSource to theTarget again.
>
>I'd expect doc 2 to be restored to theTarget, but it is not being resorted.
>
>I've tested this in 1.2 and 1.3. I don't have 1.4 setup yet, so I haven't
>tested this senario on 1.4.
>
>Am I misunderstanding how this is supposed to work, or is there a bug?
>
>Thanks!
>
>-Brad
Hey Brad,
TL;DR purge is not for normal usage, YMMV, as you’ve noticed, is really only intended for the situation when you store your Swiss Bank Account details in a doc.
If you add a 3rd node, theTardis, to this mix, and set up 2 replications, theSource->theTardis->theTarget, then the doc will re-appear on theTarget as you might expect, as theMardis won’t have knowledge of the sequence number updates that are shared knowledge between theSource & theTarget.
BTW other than curiosity, what are you trying to achieve here?
The only way I would feel comfortable achieving a purge in a *production* replicated system is to switch to a different DB, replicated off theSource, and then progressively drop each replicated node off one by one, adding the new db in one by one, including a validation doc that will not accept the purged doc id.
Finally, all endpoints and instances that keep some state (docs) need to be disinfected of the tainted doc entirely. Distributed systems are hard & confusing even on a good day. Just thinking about this in a sharded cluster gives me a headache.
I suspect I’m not alone in thinking that removing purge entirely from the API would not be a Bad Thing, as the more nodes / endpoints you add, the less likely it is to work the way you’d casually (causally?) expect.
A+
Dave