You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by eric casteleijn <er...@canonical.com> on 2009/05/26 12:11:30 UTC
Re: Document update notifications
>>> Would the comet notifications supersede the (currently implemented) db
>>> update notifications as well? Are there any estimates as to when that
>>> would come around.
>>>
>>
>> Pretty sure the idea is to replace the current system's functionality.
>> I'm not sure if that means update_notification will go away or not,
>> but from what I gather on the expected use cases it'll be a "new code
>> should use comet" type of situation. As far as estimates, I'm not
>> sure. I don't have too much knowledge on the current status, just that
>> there was work being done on some necessary bits. Keep an eye on dev@
>> and you'll be up to date as things start building up.
>
> I'm working on it and even I can't say when :) Too much stuff pops up
> that's out of my control, so I don't give completion estimates any more.
>
> But I will say it's my highest priority development task right now.
I'm sorry for asking again, but this feature (or rather, getting rid of
the update notifiers) is turning out to be more and more critical to us.
Is there *anything* we can do to help this along? If so, please let me
know, on or off list.
--
- eric casteleijn
http://www.canonical.com
Re: Document update notifications
Posted by Chris Anderson <jc...@apache.org>.
http://jchrisa.net/drl/_all_docs_by_seq also has gaps, see 371 in here.
The coolest thing that could happen to _changes right now is a
multi-part mime encoding so we could stream attachments on the same
connection as documents for true continuous replication.
On Sat, May 30, 2009 at 9:29 PM, Paul Davis <pa...@gmail.com> wrote:
> On Sat, May 30, 2009 at 11:48 PM, Randall Leeds <ra...@gmail.com> wrote:
>> On Tue, May 26, 2009 at 11:22, Chris Anderson <jc...@apache.org> wrote:
>>>
>>> There are other parts of the story we could add but haven't. For
>>> instance, it should be possible to do a long poll on a particular
>>> document URL if you are waiting for it to be modified. Also,
>>> replication should be able to run against the _changes API instead of
>>> _all_docs_by_seq, although I'm not sure that's crucial for your use
>>> case.
>>>
>>> I hope that give enough clarity for you.
>>>
>>> Chris
>>
>>
>> I just took a glance at _changes. When a compaction is performed and then a
>> fairly old sequence number is requested as the "since" sequence number, are
>> there simply holes in the seq count where old revisions used to be?
>>
>
> Unless something major has changed, yes there are holes. Ie, if a
> document update results in a document being update sequence id N, then
> you update that document again and it becomes update sequence N+1,
> then the previous update sequence (N) no longer exists.
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Document update notifications
Posted by Paul Davis <pa...@gmail.com>.
On Sat, May 30, 2009 at 11:48 PM, Randall Leeds <ra...@gmail.com> wrote:
> On Tue, May 26, 2009 at 11:22, Chris Anderson <jc...@apache.org> wrote:
>>
>> There are other parts of the story we could add but haven't. For
>> instance, it should be possible to do a long poll on a particular
>> document URL if you are waiting for it to be modified. Also,
>> replication should be able to run against the _changes API instead of
>> _all_docs_by_seq, although I'm not sure that's crucial for your use
>> case.
>>
>> I hope that give enough clarity for you.
>>
>> Chris
>
>
> I just took a glance at _changes. When a compaction is performed and then a
> fairly old sequence number is requested as the "since" sequence number, are
> there simply holes in the seq count where old revisions used to be?
>
Unless something major has changed, yes there are holes. Ie, if a
document update results in a document being update sequence id N, then
you update that document again and it becomes update sequence N+1,
then the previous update sequence (N) no longer exists.
Re: Document update notifications
Posted by Randall Leeds <ra...@gmail.com>.
On Tue, May 26, 2009 at 11:22, Chris Anderson <jc...@apache.org> wrote:
>
> There are other parts of the story we could add but haven't. For
> instance, it should be possible to do a long poll on a particular
> document URL if you are waiting for it to be modified. Also,
> replication should be able to run against the _changes API instead of
> _all_docs_by_seq, although I'm not sure that's crucial for your use
> case.
>
> I hope that give enough clarity for you.
>
> Chris
I just took a glance at _changes. When a compaction is performed and then a
fairly old sequence number is requested as the "since" sequence number, are
there simply holes in the seq count where old revisions used to be?
Re: Document update notifications
Posted by eric casteleijn <er...@canonical.com>.
Chris Anderson wrote:
> The next step on the roadmap is to give the ability to filter the
> comet notifications with a JavaScript function. This code should not
> be too hard to write but I doubt I'll have time to get to it this
> week. There are a few other people who are capable of adding the
> function, if any of them add the capability I'll be happy to review
> and apply the patch. Otherwise it should just be a matter of time
> before the comet feature is complete.
>
> There are other parts of the story we could add but haven't. For
> instance, it should be possible to do a long poll on a particular
> document URL if you are waiting for it to be modified. Also,
> replication should be able to run against the _changes API instead of
> _all_docs_by_seq, although I'm not sure that's crucial for your use
> case.
>
> I hope that give enough clarity for you.
That is very good news! I did not know that the _changes view was
already in trunk. I'll get it up and running and see if I can change our
code to start listening to it.
When you talk about filtering, does that mean taking the output it
currently gives and suppressing lines (which contain seq, doc_id and
list of revisions, if I understand correctly) or to allow emitting other
fields, or depending on the values of other fields as well?
The use case I am thinking of again is to be able to distinguish between
creation, modification and deletion of documents. Perhaps the first
could be done by inspecting revision numbers, although that may be
relying on internals too much, and I don't know what could be done to
detect whether a document was deleted.
The alternative I have seen suggested a lot, to have separate documents
that record deletion/creation events is hard to implement when the code
accessing the database is not the same on all nodes (i.e. you have no
control over some nodes, but would still like to have an idea of when
documents that were created there were first seen/deleted by the node
you do control.)
Also it concerns me that the _changes feed is now per database, rather
than per server, (although there is
https://issues.apache.org/jira/browse/COUCHDB-360 which implies a global
_changes feed, but it seems that that in turn will only include the
database name.)
Having to listen to one feed per database seems like it could take up a
lot of resources if one has a very large number of databases, which I
imagine we will.
Having to constantly query the databases for more information after each
update also seems like it would become a performance burden very soon.
This is one of the main reasons (the other being couchdb running
asynchronous, and hence hard to test, processes) that I am looking
forward to seeing the end of the update triggers as they are.
Ideally, I would like to have one _changes feed per server, with
configurable output, so that any external processing that needs to
happen can happen by one process per server, that will not have to query
into the database after every received update.
--
- eric casteleijn
http://www.canonical.com
Re: Document update notifications
Posted by Chris Anderson <jc...@apache.org>.
On Tue, May 26, 2009 at 3:11 AM, eric casteleijn
<er...@canonical.com> wrote:
>>>> Would the comet notifications supersede the (currently implemented) db
>>>> update notifications as well? Are there any estimates as to when that
>>>> would come around.
>>>>
>>>
>>> Pretty sure the idea is to replace the current system's functionality.
>>> I'm not sure if that means update_notification will go away or not,
>>> but from what I gather on the expected use cases it'll be a "new code
>>> should use comet" type of situation. As far as estimates, I'm not
>>> sure. I don't have too much knowledge on the current status, just that
>>> there was work being done on some necessary bits. Keep an eye on dev@
>>> and you'll be up to date as things start building up.
>>
>> I'm working on it and even I can't say when :) Too much stuff pops up
>> that's out of my control, so I don't give completion estimates any more.
>>
>> But I will say it's my highest priority development task right now.
>
> I'm sorry for asking again, but this feature (or rather, getting rid of the
> update notifiers) is turning out to be more and more critical to us. Is
> there *anything* we can do to help this along? If so, please let me know, on
> or off list.
>
The next step on the roadmap is to give the ability to filter the
comet notifications with a JavaScript function. This code should not
be too hard to write but I doubt I'll have time to get to it this
week. There are a few other people who are capable of adding the
function, if any of them add the capability I'll be happy to review
and apply the patch. Otherwise it should just be a matter of time
before the comet feature is complete.
There are other parts of the story we could add but haven't. For
instance, it should be possible to do a long poll on a particular
document URL if you are waiting for it to be modified. Also,
replication should be able to run against the _changes API instead of
_all_docs_by_seq, although I'm not sure that's crucial for your use
case.
I hope that give enough clarity for you.
Chris
--
Chris Anderson
http://jchrisa.net
http://couch.io