_changes and filters not working in harmony?

[Timothy Wood <tj...@omnigroup.com>]

I’ve been playing with the _changes support in concert with the filter parameter. I’m seeing some odd results, though.

In all these cases, my filter function looks like:

function(doc, req) {
  if (doc.type == "item" && doc["_attachments"]) {
    return true;
  }
  return false;
}


1) If I query for changes with a filter like:

	http://status-board:5984/status-board/_changes?feed=continuous&filter=app/items

I’ll see my test documents (that match the filter) come back. Great. But then if I go and delete a document using Futon, no “_deleted”:true change comes back. If I leave off the filter, I do get a _deleted entry in the changes feed.

2) If I edit the document in Futon (by deleting its last attachment) so that it no longer matches the filter, I don’t get a _deleted entry. I would expect to get one since the change feed reported the document to me (it matched the filter) and then it was edited to not match the filter. For an observer following a filtered feed, this delete seems pretty crucial (if you wanted to do a filtered replication, for example).

If I start a an unfiltered _changes feed and then do an insert of a document followed by adding an attachment, I get two lines in the _changes feed. But, if I use the filter, I only get one. That is, unlike the delete case, the filter doesn’t note the document until the document matches the filter. Probably a happy accident, but the mirror case (edit to not match the filter) should be symmetric and issue a delete change.

In my particular use case, I can probably just drop the filter on the server side and do it in the client. Is there something wrong in my use of _changes with a filter? I don’t see any discussion of this in the book or on the mailing list, but maybe I missed it.

Thanks!

-tim

Re: _changes and filters not working in harmony?

[Matt Goodall <ma...@gmail.com>]

On 5 February 2011 23:59, Timothy Wood <tj...@omnigroup.com> wrote:
>
> I’ve been playing with the _changes support in concert with the filter parameter. I’m seeing some odd results, though.
>
> In all these cases, my filter function looks like:
>
> function(doc, req) {
>  if (doc.type == "item" && doc["_attachments"]) {
>    return true;
>  }
>  return false;
> }
>
>
> 1) If I query for changes with a filter like:
>
>        http://status-board:5984/status-board/_changes?feed=continuous&filter=app/items
>
> I’ll see my test documents (that match the filter) come back. Great. But then if I go and delete a document using Futon, no “_deleted”:true change comes back. If I leave off the filter, I do get a _deleted entry in the changes feed.

Unfortunately, a deleted document does not retain existing attributes.
A deleted doc is basically just a stub containing _id, _rev and
_deleted attributes.

Personally, I think it would be very useful to retain a document's
existing attributes when deleted, i.e. add a _deleted attribute and
leave everything else alone.

>
> 2) If I edit the document in Futon (by deleting its last attachment) so that it no longer matches the filter, I don’t get a _deleted entry. I would expect to get one since the change feed reported the document to me (it matched the filter) and then it was edited to not match the filter. For an observer following a filtered feed, this delete seems pretty crucial (if you wanted to do a filtered replication, for example).

I don't really understand why you would expect a _deleted row in the
changes feed here. The document has not been deleted, just changed,
and your filter function explicitly says it's not interested in
documents with no attachments.

You should remove the test for doc["_attachments"] and handle
attachments vs no attachments in your change feed handled.

>
> If I start a an unfiltered _changes feed and then do an insert of a document followed by adding an attachment, I get two lines in the _changes feed. But, if I use the filter, I only get one. That is, unlike the delete case, the filter doesn’t note the document until the document matches the filter. Probably a happy accident, but the mirror case (edit to not match the filter) should be symmetric and issue a delete change.

It's really not a happy accident - it's exactly what your filter
function tells CouchDB to do.

>
> In my particular use case, I can probably just drop the filter on the server side and do it in the client. Is there something wrong in my use of _changes with a filter? I don’t see any discussion of this in the book or on the mailing list, but maybe I missed it.

You don't need to drop the filter func but you might have to
compromise a little. Change it to the following:

function(doc, req) {
  return doc._deleted || doc.type == "item";
}

You can then:

* Handle attachments vs no attachments correctly
* Clean up after a document gets deleted.

Note that you will see *all* deleted documents and not just documents
that used to have an item: "type" attribute. Admittedly, that's not
ideal but, in my experience, not a massive problem either.

Hope that helps.

- Matt

Re: _changes and filters not working in harmony?

[Matt Goodall <ma...@gmail.com>]

On 6 February 2011 10:21, Robert Newson <ro...@gmail.com> wrote:
> I guess I'm missing something here, but since your function will
> return false for a deleted document, what's the surprise?
>
> A deleted document is exactly this;
>
> {"_id":"foo","_deleted":true}
>
> It has no type and no attachments, it's therefore filtered out.
>
> if you add 'if (doc._deleted) { return true; }' you'll see the delete
> but, as Randall points out, you'll see all deletes. To solve that, we
> might want to pass either the previous doc *or* simply add the
> _deleted:true field when we delete, meaning that your filter would
> work as-is.

Hmm, having just said that I'd like to see the attributes of document
retained when it's deleted I'm not sure it's as useful as it initially
sounds.

When you're processing a row in a _changes feed there's really no
guarantee that the version of the document you saw last is anything
like the version when it's deleted. In fact, it's entirely possible
the deleted version is the first time you see a document!

So, it could be quite misleading to people writing _changes feed
handlers if the old data is made available.

- Matt

>
> B.
>
> On Sun, Feb 6, 2011 at 12:36 AM, Randall Leeds <ra...@gmail.com> wrote:
>> I don't think there's anything wrong with what you're doing here, but I
>> don't know the code too well.
>> It sounds like what maybe we need is to pass the old document as a third
>> parameter or something.
>>
>> On Sat, Feb 5, 2011 at 15:59, Timothy Wood <tj...@omnigroup.com> wrote:
>>
>>>
>>> I’ve been playing with the _changes support in concert with the filter
>>> parameter. I’m seeing some odd results, though.
>>>
>>> In all these cases, my filter function looks like:
>>>
>>> function(doc, req) {
>>>  if (doc.type == "item" && doc["_attachments"]) {
>>>    return true;
>>>  }
>>>  return false;
>>> }
>>>
>>>
>>> 1) If I query for changes with a filter like:
>>>
>>>
>>> http://status-board:5984/status-board/_changes?feed=continuous&filter=app/items
>>>
>>> I’ll see my test documents (that match the filter) come back. Great. But
>>> then if I go and delete a document using Futon, no “_deleted”:true change
>>> comes back. If I leave off the filter, I do get a _deleted entry in the
>>> changes feed.
>>>
>>> 2) If I edit the document in Futon (by deleting its last attachment) so
>>> that it no longer matches the filter, I don’t get a _deleted entry. I would
>>> expect to get one since the change feed reported the document to me (it
>>> matched the filter) and then it was edited to not match the filter. For an
>>> observer following a filtered feed, this delete seems pretty crucial (if you
>>> wanted to do a filtered replication, for example).
>>>
>>> If I start a an unfiltered _changes feed and then do an insert of a
>>> document followed by adding an attachment, I get two lines in the _changes
>>> feed. But, if I use the filter, I only get one. That is, unlike the delete
>>> case, the filter doesn’t note the document until the document matches the
>>> filter. Probably a happy accident, but the mirror case (edit to not match
>>> the filter) should be symmetric and issue a delete change.
>>>
>>> In my particular use case, I can probably just drop the filter on the
>>> server side and do it in the client. Is there something wrong in my use of
>>> _changes with a filter? I don’t see any discussion of this in the book or on
>>> the mailing list, but maybe I missed it.
>>>
>>> Thanks!
>>>
>>> -tim
>>>
>>>
>>
>

Re: _changes and filters not working in harmony?

[Robert Newson <ro...@gmail.com>]

I guess I'm missing something here, but since your function will
return false for a deleted document, what's the surprise?

A deleted document is exactly this;

{"_id":"foo","_deleted":true}

It has no type and no attachments, it's therefore filtered out.

if you add 'if (doc._deleted) { return true; }' you'll see the delete
but, as Randall points out, you'll see all deletes. To solve that, we
might want to pass either the previous doc *or* simply add the
_deleted:true field when we delete, meaning that your filter would
work as-is.

B.

On Sun, Feb 6, 2011 at 12:36 AM, Randall Leeds <ra...@gmail.com> wrote:
> I don't think there's anything wrong with what you're doing here, but I
> don't know the code too well.
> It sounds like what maybe we need is to pass the old document as a third
> parameter or something.
>
> On Sat, Feb 5, 2011 at 15:59, Timothy Wood <tj...@omnigroup.com> wrote:
>
>>
>> I’ve been playing with the _changes support in concert with the filter
>> parameter. I’m seeing some odd results, though.
>>
>> In all these cases, my filter function looks like:
>>
>> function(doc, req) {
>>  if (doc.type == "item" && doc["_attachments"]) {
>>    return true;
>>  }
>>  return false;
>> }
>>
>>
>> 1) If I query for changes with a filter like:
>>
>>
>> http://status-board:5984/status-board/_changes?feed=continuous&filter=app/items
>>
>> I’ll see my test documents (that match the filter) come back. Great. But
>> then if I go and delete a document using Futon, no “_deleted”:true change
>> comes back. If I leave off the filter, I do get a _deleted entry in the
>> changes feed.
>>
>> 2) If I edit the document in Futon (by deleting its last attachment) so
>> that it no longer matches the filter, I don’t get a _deleted entry. I would
>> expect to get one since the change feed reported the document to me (it
>> matched the filter) and then it was edited to not match the filter. For an
>> observer following a filtered feed, this delete seems pretty crucial (if you
>> wanted to do a filtered replication, for example).
>>
>> If I start a an unfiltered _changes feed and then do an insert of a
>> document followed by adding an attachment, I get two lines in the _changes
>> feed. But, if I use the filter, I only get one. That is, unlike the delete
>> case, the filter doesn’t note the document until the document matches the
>> filter. Probably a happy accident, but the mirror case (edit to not match
>> the filter) should be symmetric and issue a delete change.
>>
>> In my particular use case, I can probably just drop the filter on the
>> server side and do it in the client. Is there something wrong in my use of
>> _changes with a filter? I don’t see any discussion of this in the book or on
>> the mailing list, but maybe I missed it.
>>
>> Thanks!
>>
>> -tim
>>
>>
>

Re: _changes and filters not working in harmony?

[Randall Leeds <ra...@gmail.com>]

I don't think there's anything wrong with what you're doing here, but I
don't know the code too well.
It sounds like what maybe we need is to pass the old document as a third
parameter or something.

On Sat, Feb 5, 2011 at 15:59, Timothy Wood <tj...@omnigroup.com> wrote:

>
> I’ve been playing with the _changes support in concert with the filter
> parameter. I’m seeing some odd results, though.
>
> In all these cases, my filter function looks like:
>
> function(doc, req) {
>  if (doc.type == "item" && doc["_attachments"]) {
>    return true;
>  }
>  return false;
> }
>
>
> 1) If I query for changes with a filter like:
>
>
> http://status-board:5984/status-board/_changes?feed=continuous&filter=app/items
>
> I’ll see my test documents (that match the filter) come back. Great. But
> then if I go and delete a document using Futon, no “_deleted”:true change
> comes back. If I leave off the filter, I do get a _deleted entry in the
> changes feed.
>
> 2) If I edit the document in Futon (by deleting its last attachment) so
> that it no longer matches the filter, I don’t get a _deleted entry. I would
> expect to get one since the change feed reported the document to me (it
> matched the filter) and then it was edited to not match the filter. For an
> observer following a filtered feed, this delete seems pretty crucial (if you
> wanted to do a filtered replication, for example).
>
> If I start a an unfiltered _changes feed and then do an insert of a
> document followed by adding an attachment, I get two lines in the _changes
> feed. But, if I use the filter, I only get one. That is, unlike the delete
> case, the filter doesn’t note the document until the document matches the
> filter. Probably a happy accident, but the mirror case (edit to not match
> the filter) should be symmetric and issue a delete change.
>
> In my particular use case, I can probably just drop the filter on the
> server side and do it in the client. Is there something wrong in my use of
> _changes with a filter? I don’t see any discussion of this in the book or on
> the mailing list, but maybe I missed it.
>
> Thanks!
>
> -tim
>
>