You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Stefan Klein <st...@gmail.com> on 2017/10/27 16:34:22 UTC
Documentation about the changes "seq" in couchdb 2.x?
Hi,
the changes feed Api documentation states that (simplified) the order
of changes is not guaranteed and the same seq might show up multiple
times.
Is there some more documentation on the sequence, the since parameter,
how to make sure to actually see the changes I'm interested in?
I like to read first before i ask specific questions - if any are left.
regards,
Stefan
Re: Documentation about the changes "seq" in couchdb 2.x?
Posted by Robert Samuel Newson <rn...@apache.org>.
idempotent processing of changes is not a recommendation, it's a requirement. :)
for non-continuous changes responses, you will only see each document once.
B.
> On 2 Nov 2017, at 14:24, Stefan Klein <st...@gmail.com> wrote:
>
> Hi,
>
> i agree, idempotent processing is the best option, when I wrote the
> previous mail I implied if idempotent processing is not possible but
> didn't write it.
>
> One more question came up:
> In 1.x the changes feed did only contain the latest change/revision of
> a single document.
> Is this still true for 2.x or could it even happen to see a single
> document twice in the feed, maybe even in the wrong order?
>
> thanks,
> Stefan
>
>
> 2017-10-30 18:44 GMT+01:00 Carlos Alonso <ca...@cabify.com>:
>> IMHO I think the best approach to this is to try to have an idempotent
>> processing mechanism. I think this will be the strongest approach as
>> flagging docs as processed in a distributed system has a few interesting
>> edge cases in which the update is lost and if that is harmful to your
>> system it is something I'd definitely protect against.
>>
>> Flagging docs as processed also makes documents update more complicated, as
>> you'd need to remove the flag in order to be able to process the new
>> version...
>>
>> Also, idempotent processing is Robert's recommendation on his writeup.
>> Thanks for that explanation, really clarifying!
>>
>> Regards
>>
>> On Mon, Oct 30, 2017 at 4:16 PM Stefan Klein <st...@gmail.com> wrote:
>>
>>> Hi Jan,
>>>
>>> thank you for the fast answer, and thanks to Robert for the writeup.
>>>
>>> I do have do process changes asynchronously, so a younger change can
>>> be already processed before an older change is ready.
>>> With 1.x i interpreted the sequence as an ID of the change, with this
>>> i could filter the already processed changes and determine the since
>>> param to be set to the lowest sequence which wasn't processed yet.
>>>
>>> With 2.x this isn't possible at all, the sequence for the same change
>>> (doc@rev) may differ between calls to _changes, we might see changes
>>> multiple times. So the only reasonable way i see is to write back to
>>> the processed document, to persist the "is processed" information in
>>> the document itself. (If for this particular case it matters whether
>>> the same change is processed multiple times).
>>>
>>> Thanks again,
>>> Stefan
>>>
>>>
>>>
>>>
>>> 2017-10-27 21:08 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>>> :blushes:
>>>>
>>>>> On 27 Oct 2017, at 18:53, Jan Lehnardt <ja...@apache.org> wrote:
>>>>>
>>>>> Hi Stefan,
>>>>>
>>>>> this is about as comprehensive as it gets, curtesy of Bob Newson:
>>>>>
>>>>>
>>> https://lists.apache.org/thread.html/aaf94d3e1e67155912e2d68cd584e64366ed34f8f47578a60731afaf@%3Cuser.couchdb.apache.org%3E
>>>>>
>>>>> Context thread:
>>> https://lists.apache.org/thread.html/d2f1cf167f931ccb62552b681f44559168755d37b0f51ec1982edbbf@%3Cuser.couchdb.apache.org%3E
>>>>>
>>>>> Best
>>>>> Jan
>>>>> --
>>>>>
>>>>>> On 27. Oct 2017, at 18:34, Stefan Klein <st...@gmail.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> the changes feed Api documentation states that (simplified) the order
>>>>>> of changes is not guaranteed and the same seq might show up multiple
>>>>>> times.
>>>>>>
>>>>>> Is there some more documentation on the sequence, the since parameter,
>>>>>> how to make sure to actually see the changes I'm interested in?
>>>>>> I like to read first before i ask specific questions - if any are left.
>>>>>>
>>>>>> regards,
>>>>>> Stefan
>>>>>
>>>>> --
>>>>> Professional Support for Apache CouchDB:
>>>>> https://neighbourhood.ie/couchdb-support/
>>>>>
>>>>
>>>
>> --
>> [image: Cabify - Your private Driver] <http://www.cabify.com/>
>>
>> *Carlos Alonso*
>> Data Engineer
>> Madrid, Spain
>>
>> carlos.alonso@cabify.com
>>
>> [image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
>> <http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image:
>> Linkedin] <https://www.linkedin.com/in/mrcalonso>
>>
>> --
>> Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su
>> destinatario, pudiendo contener información confidencial sometida a secreto
>> profesional. No está permitida su reproducción o distribución sin la
>> autorización expresa de Cabify. Si usted no es el destinatario final por
>> favor elimínelo e infórmenos por esta vía.
>>
>> This message and any attached file are intended exclusively for the
>> addressee, and it may be confidential. You are not allowed to copy or
>> disclose it without Cabify's prior written authorization. If you are not
>> the intended recipient please delete it from your system and notify us by
>> e-mail.
Re: Documentation about the changes "seq" in couchdb 2.x?
Posted by Stefan Klein <st...@gmail.com>.
Hi,
i agree, idempotent processing is the best option, when I wrote the
previous mail I implied if idempotent processing is not possible but
didn't write it.
One more question came up:
In 1.x the changes feed did only contain the latest change/revision of
a single document.
Is this still true for 2.x or could it even happen to see a single
document twice in the feed, maybe even in the wrong order?
thanks,
Stefan
2017-10-30 18:44 GMT+01:00 Carlos Alonso <ca...@cabify.com>:
> IMHO I think the best approach to this is to try to have an idempotent
> processing mechanism. I think this will be the strongest approach as
> flagging docs as processed in a distributed system has a few interesting
> edge cases in which the update is lost and if that is harmful to your
> system it is something I'd definitely protect against.
>
> Flagging docs as processed also makes documents update more complicated, as
> you'd need to remove the flag in order to be able to process the new
> version...
>
> Also, idempotent processing is Robert's recommendation on his writeup.
> Thanks for that explanation, really clarifying!
>
> Regards
>
> On Mon, Oct 30, 2017 at 4:16 PM Stefan Klein <st...@gmail.com> wrote:
>
>> Hi Jan,
>>
>> thank you for the fast answer, and thanks to Robert for the writeup.
>>
>> I do have do process changes asynchronously, so a younger change can
>> be already processed before an older change is ready.
>> With 1.x i interpreted the sequence as an ID of the change, with this
>> i could filter the already processed changes and determine the since
>> param to be set to the lowest sequence which wasn't processed yet.
>>
>> With 2.x this isn't possible at all, the sequence for the same change
>> (doc@rev) may differ between calls to _changes, we might see changes
>> multiple times. So the only reasonable way i see is to write back to
>> the processed document, to persist the "is processed" information in
>> the document itself. (If for this particular case it matters whether
>> the same change is processed multiple times).
>>
>> Thanks again,
>> Stefan
>>
>>
>>
>>
>> 2017-10-27 21:08 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>> > :blushes:
>> >
>> >> On 27 Oct 2017, at 18:53, Jan Lehnardt <ja...@apache.org> wrote:
>> >>
>> >> Hi Stefan,
>> >>
>> >> this is about as comprehensive as it gets, curtesy of Bob Newson:
>> >>
>> >>
>> https://lists.apache.org/thread.html/aaf94d3e1e67155912e2d68cd584e64366ed34f8f47578a60731afaf@%3Cuser.couchdb.apache.org%3E
>> >>
>> >> Context thread:
>> https://lists.apache.org/thread.html/d2f1cf167f931ccb62552b681f44559168755d37b0f51ec1982edbbf@%3Cuser.couchdb.apache.org%3E
>> >>
>> >> Best
>> >> Jan
>> >> --
>> >>
>> >>> On 27. Oct 2017, at 18:34, Stefan Klein <st...@gmail.com> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> the changes feed Api documentation states that (simplified) the order
>> >>> of changes is not guaranteed and the same seq might show up multiple
>> >>> times.
>> >>>
>> >>> Is there some more documentation on the sequence, the since parameter,
>> >>> how to make sure to actually see the changes I'm interested in?
>> >>> I like to read first before i ask specific questions - if any are left.
>> >>>
>> >>> regards,
>> >>> Stefan
>> >>
>> >> --
>> >> Professional Support for Apache CouchDB:
>> >> https://neighbourhood.ie/couchdb-support/
>> >>
>> >
>>
> --
> [image: Cabify - Your private Driver] <http://www.cabify.com/>
>
> *Carlos Alonso*
> Data Engineer
> Madrid, Spain
>
> carlos.alonso@cabify.com
>
> [image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
> <http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image:
> Linkedin] <https://www.linkedin.com/in/mrcalonso>
>
> --
> Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su
> destinatario, pudiendo contener información confidencial sometida a secreto
> profesional. No está permitida su reproducción o distribución sin la
> autorización expresa de Cabify. Si usted no es el destinatario final por
> favor elimínelo e infórmenos por esta vía.
>
> This message and any attached file are intended exclusively for the
> addressee, and it may be confidential. You are not allowed to copy or
> disclose it without Cabify's prior written authorization. If you are not
> the intended recipient please delete it from your system and notify us by
> e-mail.
Re: Documentation about the changes "seq" in couchdb 2.x?
Posted by Carlos Alonso <ca...@cabify.com>.
IMHO I think the best approach to this is to try to have an idempotent
processing mechanism. I think this will be the strongest approach as
flagging docs as processed in a distributed system has a few interesting
edge cases in which the update is lost and if that is harmful to your
system it is something I'd definitely protect against.
Flagging docs as processed also makes documents update more complicated, as
you'd need to remove the flag in order to be able to process the new
version...
Also, idempotent processing is Robert's recommendation on his writeup.
Thanks for that explanation, really clarifying!
Regards
On Mon, Oct 30, 2017 at 4:16 PM Stefan Klein <st...@gmail.com> wrote:
> Hi Jan,
>
> thank you for the fast answer, and thanks to Robert for the writeup.
>
> I do have do process changes asynchronously, so a younger change can
> be already processed before an older change is ready.
> With 1.x i interpreted the sequence as an ID of the change, with this
> i could filter the already processed changes and determine the since
> param to be set to the lowest sequence which wasn't processed yet.
>
> With 2.x this isn't possible at all, the sequence for the same change
> (doc@rev) may differ between calls to _changes, we might see changes
> multiple times. So the only reasonable way i see is to write back to
> the processed document, to persist the "is processed" information in
> the document itself. (If for this particular case it matters whether
> the same change is processed multiple times).
>
> Thanks again,
> Stefan
>
>
>
>
> 2017-10-27 21:08 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
> > :blushes:
> >
> >> On 27 Oct 2017, at 18:53, Jan Lehnardt <ja...@apache.org> wrote:
> >>
> >> Hi Stefan,
> >>
> >> this is about as comprehensive as it gets, curtesy of Bob Newson:
> >>
> >>
> https://lists.apache.org/thread.html/aaf94d3e1e67155912e2d68cd584e64366ed34f8f47578a60731afaf@%3Cuser.couchdb.apache.org%3E
> >>
> >> Context thread:
> https://lists.apache.org/thread.html/d2f1cf167f931ccb62552b681f44559168755d37b0f51ec1982edbbf@%3Cuser.couchdb.apache.org%3E
> >>
> >> Best
> >> Jan
> >> --
> >>
> >>> On 27. Oct 2017, at 18:34, Stefan Klein <st...@gmail.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> the changes feed Api documentation states that (simplified) the order
> >>> of changes is not guaranteed and the same seq might show up multiple
> >>> times.
> >>>
> >>> Is there some more documentation on the sequence, the since parameter,
> >>> how to make sure to actually see the changes I'm interested in?
> >>> I like to read first before i ask specific questions - if any are left.
> >>>
> >>> regards,
> >>> Stefan
> >>
> >> --
> >> Professional Support for Apache CouchDB:
> >> https://neighbourhood.ie/couchdb-support/
> >>
> >
>
--
[image: Cabify - Your private Driver] <http://www.cabify.com/>
*Carlos Alonso*
Data Engineer
Madrid, Spain
carlos.alonso@cabify.com
[image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
<http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image:
Linkedin] <https://www.linkedin.com/in/mrcalonso>
--
Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su
destinatario, pudiendo contener información confidencial sometida a secreto
profesional. No está permitida su reproducción o distribución sin la
autorización expresa de Cabify. Si usted no es el destinatario final por
favor elimínelo e infórmenos por esta vía.
This message and any attached file are intended exclusively for the
addressee, and it may be confidential. You are not allowed to copy or
disclose it without Cabify's prior written authorization. If you are not
the intended recipient please delete it from your system and notify us by
e-mail.
Re: Documentation about the changes "seq" in couchdb 2.x?
Posted by Stefan Klein <st...@gmail.com>.
Hi Jan,
thank you for the fast answer, and thanks to Robert for the writeup.
I do have do process changes asynchronously, so a younger change can
be already processed before an older change is ready.
With 1.x i interpreted the sequence as an ID of the change, with this
i could filter the already processed changes and determine the since
param to be set to the lowest sequence which wasn't processed yet.
With 2.x this isn't possible at all, the sequence for the same change
(doc@rev) may differ between calls to _changes, we might see changes
multiple times. So the only reasonable way i see is to write back to
the processed document, to persist the "is processed" information in
the document itself. (If for this particular case it matters whether
the same change is processed multiple times).
Thanks again,
Stefan
2017-10-27 21:08 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
> :blushes:
>
>> On 27 Oct 2017, at 18:53, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> Hi Stefan,
>>
>> this is about as comprehensive as it gets, curtesy of Bob Newson:
>>
>> https://lists.apache.org/thread.html/aaf94d3e1e67155912e2d68cd584e64366ed34f8f47578a60731afaf@%3Cuser.couchdb.apache.org%3E
>>
>> Context thread: https://lists.apache.org/thread.html/d2f1cf167f931ccb62552b681f44559168755d37b0f51ec1982edbbf@%3Cuser.couchdb.apache.org%3E
>>
>> Best
>> Jan
>> --
>>
>>> On 27. Oct 2017, at 18:34, Stefan Klein <st...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> the changes feed Api documentation states that (simplified) the order
>>> of changes is not guaranteed and the same seq might show up multiple
>>> times.
>>>
>>> Is there some more documentation on the sequence, the since parameter,
>>> how to make sure to actually see the changes I'm interested in?
>>> I like to read first before i ask specific questions - if any are left.
>>>
>>> regards,
>>> Stefan
>>
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>>
>
Re: Documentation about the changes "seq" in couchdb 2.x?
Posted by Robert Samuel Newson <rn...@apache.org>.
:blushes:
> On 27 Oct 2017, at 18:53, Jan Lehnardt <ja...@apache.org> wrote:
>
> Hi Stefan,
>
> this is about as comprehensive as it gets, curtesy of Bob Newson:
>
> https://lists.apache.org/thread.html/aaf94d3e1e67155912e2d68cd584e64366ed34f8f47578a60731afaf@%3Cuser.couchdb.apache.org%3E
>
> Context thread: https://lists.apache.org/thread.html/d2f1cf167f931ccb62552b681f44559168755d37b0f51ec1982edbbf@%3Cuser.couchdb.apache.org%3E
>
> Best
> Jan
> --
>
>> On 27. Oct 2017, at 18:34, Stefan Klein <st...@gmail.com> wrote:
>>
>> Hi,
>>
>> the changes feed Api documentation states that (simplified) the order
>> of changes is not guaranteed and the same seq might show up multiple
>> times.
>>
>> Is there some more documentation on the sequence, the since parameter,
>> how to make sure to actually see the changes I'm interested in?
>> I like to read first before i ask specific questions - if any are left.
>>
>> regards,
>> Stefan
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
Re: Documentation about the changes "seq" in couchdb 2.x?
Posted by Jan Lehnardt <ja...@apache.org>.
Hi Stefan,
this is about as comprehensive as it gets, curtesy of Bob Newson:
https://lists.apache.org/thread.html/aaf94d3e1e67155912e2d68cd584e64366ed34f8f47578a60731afaf@%3Cuser.couchdb.apache.org%3E
Context thread: https://lists.apache.org/thread.html/d2f1cf167f931ccb62552b681f44559168755d37b0f51ec1982edbbf@%3Cuser.couchdb.apache.org%3E
Best
Jan
--
> On 27. Oct 2017, at 18:34, Stefan Klein <st...@gmail.com> wrote:
>
> Hi,
>
> the changes feed Api documentation states that (simplified) the order
> of changes is not guaranteed and the same seq might show up multiple
> times.
>
> Is there some more documentation on the sequence, the since parameter,
> how to make sure to actually see the changes I'm interested in?
> I like to read first before i ask specific questions - if any are left.
>
> regards,
> Stefan
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/