You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Benoit Chesneau (Created) (JIRA)" <ji...@apache.org> on 2012/02/03 00:06:53 UTC
[jira] [Created] (COUCHDB-1398) improve view filtering in changes
improve view filtering in changes
---------------------------------
Key: COUCHDB-1398
URL: https://issues.apache.org/jira/browse/COUCHDB-1398
Project: CouchDB
Issue Type: Improvement
Components: View Server Support
Affects Versions: 2.0, 1.3
Reporter: Benoit Chesneau
Improve the native view filter `_view` support by really using view index. This patches add following features
- small refactoring: create the couch_httpd_changes modules, to put all the changes http support in its own module instead having it in couch_httpd_db.
- add the `view_updated` event when a view index is updated : {view_updated, {DbName, IndexName}}
- start the feed using results in the view index instead of all the db index
- only react on view index changes.
For now next changes are still get using the couch_db:changes_since function and passing the map function to the results. It could be improved if we had a by_seq btree in the view index too. Other way may be to skip a number of the documents already processed. Not sure it would be faster. Thoughts ?
The branch couch_view_changes in my repo contains preliminary support:
https://github.com/benoitc/couchdb/tree/couch_view_changes
Diff:
https://github.com/benoitc/couchdb/compare/master...couch_view_changes
To use it, use the native filter named _view which take the parameter view=DesignName/Viewname
eg:
http://server/db/_changes?feed=continuous&heartbeat=true&filter=_view&view=DesignName/SomeView
It has also an interresting side effect: on each db updates the view index refresh is triggered so view updates are triggered. Maybe we could introduce an optionnal parameter to not trigger them though?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1398) improve view filtering in changes
Posted by "Benoit Chesneau (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benoit Chesneau updated COUCHDB-1398:
-------------------------------------
Attachment: 0007-useless-info.patch
0006-immprove-view-filter.patch
0005-add-view_updated-event.patch
0004-This-wrapper-is-useless-somehow-split-the-code-in-a-.patch
0003-fix-indent.patch
0002-initial-step-move-the-code-from-couch_httpd_db-to-co.patch
0001-white-spaces.patch
First set of patches
> improve view filtering in changes
> ---------------------------------
>
> Key: COUCHDB-1398
> URL: https://issues.apache.org/jira/browse/COUCHDB-1398
> Project: CouchDB
> Issue Type: Improvement
> Components: View Server Support
> Affects Versions: 2.0, 1.3
> Reporter: Benoit Chesneau
> Labels: changes, view
> Attachments: 0001-white-spaces.patch, 0002-initial-step-move-the-code-from-couch_httpd_db-to-co.patch, 0003-fix-indent.patch, 0004-This-wrapper-is-useless-somehow-split-the-code-in-a-.patch, 0005-add-view_updated-event.patch, 0006-immprove-view-filter.patch, 0007-useless-info.patch
>
>
> Improve the native view filter `_view` support by really using view index. This patches add following features
> - small refactoring: create the couch_httpd_changes modules, to put all the changes http support in its own module instead having it in couch_httpd_db.
> - add the `view_updated` event when a view index is updated : {view_updated, {DbName, IndexName}}
> - start the feed using results in the view index instead of all the db index
> - only react on view index changes.
> For now next changes are still get using the couch_db:changes_since function and passing the map function to the results. It could be improved if we had a by_seq btree in the view index too. Other way may be to skip a number of the documents already processed. Not sure it would be faster. Thoughts ?
> The branch couch_view_changes in my repo contains preliminary support:
> https://github.com/benoitc/couchdb/tree/couch_view_changes
> Diff:
> https://github.com/benoitc/couchdb/compare/master...couch_view_changes
> To use it, use the native filter named _view which take the parameter view=DesignName/Viewname
> eg:
> http://server/db/_changes?feed=continuous&heartbeat=true&filter=_view&view=DesignName/SomeView
> It has also an interresting side effect: on each db updates the view index refresh is triggered so view updates are triggered. Maybe we could introduce an optionnal parameter to not trigger them though?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1398) improve view filtering in changes
Posted by "Benoit Chesneau (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207657#comment-13207657 ]
Benoit Chesneau commented on COUCHDB-1398:
------------------------------------------
bump.
> improve view filtering in changes
> ---------------------------------
>
> Key: COUCHDB-1398
> URL: https://issues.apache.org/jira/browse/COUCHDB-1398
> Project: CouchDB
> Issue Type: Improvement
> Components: View Server Support
> Affects Versions: 2.0, 1.3
> Reporter: Benoit Chesneau
> Labels: changes, view
> Attachments: 0001-white-spaces.patch, 0002-initial-step-move-the-code-from-couch_httpd_db-to-co.patch, 0003-fix-indent.patch, 0004-This-wrapper-is-useless-somehow-split-the-code-in-a-.patch, 0005-add-view_updated-event.patch, 0006-immprove-view-filter.patch, 0007-useless-info.patch, 0008-whitespaces.patch, 0009-handle-native-filters-in-views.patch
>
>
> Improve the native view filter `_view` support by really using view index. This patches add following features
> - small refactoring: create the couch_httpd_changes modules, to put all the changes http support in its own module instead having it in couch_httpd_db.
> - add the `view_updated` event when a view index is updated : {view_updated, {DbName, IndexName}}
> - start the feed using results in the view index instead of all the db index
> - only react on view index changes.
> For now next changes are still get using the couch_db:changes_since function and passing the map function to the results. It could be improved if we had a by_seq btree in the view index too. Other way may be to skip a number of the documents already processed. Not sure it would be faster. Thoughts ?
> The branch couch_view_changes in my repo contains preliminary support:
> https://github.com/benoitc/couchdb/tree/couch_view_changes
> Diff:
> https://github.com/benoitc/couchdb/compare/master...couch_view_changes
> To use it, use the native filter named _view which take the parameter view=DesignName/Viewname
> eg:
> http://server/db/_changes?feed=continuous&heartbeat=true&filter=_view&view=DesignName/SomeView
> It has also an interresting side effect: on each db updates the view index refresh is triggered so view updates are triggered. Maybe we could introduce an optionnal parameter to not trigger them though?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1398) improve view filtering in changes
Posted by "Benoit Chesneau (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208441#comment-13208441 ]
Benoit Chesneau commented on COUCHDB-1398:
------------------------------------------
Hi Filipe,
Thanks for the review, I will address issues later. Just quick comments :
1) I will fix that. I prefer this indentation to the other since it distinct the fun from other level. But will comply with common style, np :)
2) I thought opening the ddoc each time was enough to handle the ddoc changes. Isn't raising an exception enough to stop the feed? But of course checking if the ddoc has changed may be better at this point. I will have a look. Also errors should be probably catched.
3) In fact I'm eagerly waiting for an event hub in couch. Where you can publish any event and subscribe too it. I started to use couch_event from @davisp but stopped at some point since I wasn't sure it was the correct solution. To minimize the code base, I just reuse the current event handler. I've no strong opinion on it. Do you think it's really needed?
4) Manual replications tests worked. The `_view` filter is already tested. I will add tests to address cases where you already have doc, when ddoc disapear & such.
Again thanks for the review.
> improve view filtering in changes
> ---------------------------------
>
> Key: COUCHDB-1398
> URL: https://issues.apache.org/jira/browse/COUCHDB-1398
> Project: CouchDB
> Issue Type: Improvement
> Components: View Server Support
> Affects Versions: 2.0, 1.3
> Reporter: Benoit Chesneau
> Labels: changes, view
> Attachments: 0001-white-spaces.patch, 0002-initial-step-move-the-code-from-couch_httpd_db-to-co.patch, 0003-fix-indent.patch, 0004-This-wrapper-is-useless-somehow-split-the-code-in-a-.patch, 0005-add-view_updated-event.patch, 0006-immprove-view-filter.patch, 0007-useless-info.patch, 0008-whitespaces.patch, 0009-handle-native-filters-in-views.patch
>
>
> Improve the native view filter `_view` support by really using view index. This patches add following features
> - small refactoring: create the couch_httpd_changes modules, to put all the changes http support in its own module instead having it in couch_httpd_db.
> - add the `view_updated` event when a view index is updated : {view_updated, {DbName, IndexName}}
> - start the feed using results in the view index instead of all the db index
> - only react on view index changes.
> For now next changes are still get using the couch_db:changes_since function and passing the map function to the results. It could be improved if we had a by_seq btree in the view index too. Other way may be to skip a number of the documents already processed. Not sure it would be faster. Thoughts ?
> The branch couch_view_changes in my repo contains preliminary support:
> https://github.com/benoitc/couchdb/tree/couch_view_changes
> Diff:
> https://github.com/benoitc/couchdb/compare/master...couch_view_changes
> To use it, use the native filter named _view which take the parameter view=DesignName/Viewname
> eg:
> http://server/db/_changes?feed=continuous&heartbeat=true&filter=_view&view=DesignName/SomeView
> It has also an interresting side effect: on each db updates the view index refresh is triggered so view updates are triggered. Maybe we could introduce an optionnal parameter to not trigger them though?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1398) improve view filtering in changes
Posted by "Filipe Manana (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208465#comment-13208465 ]
Filipe Manana commented on COUCHDB-1398:
----------------------------------------
Benôit,
for 1) - Sure, no probem.
for 2) - That's right. It will work when the ddoc signature changes - my comment there was if continuing to send changes after that happens is ok or not for applications (since the view definition might have changed radically) - it's probably ok for many applications. I don't have strong opinion here. If no one objects it can remain like that imho.
Now for the the case the ddoc is deleted (and never recreated again), the client will hang forever. The process spawned with spawn/1 will terminate with a reason different from 'normal' (with a thow {not_found, deleted}) but the parent will never know that, because it's not linked to it or monitoring it, so it will hang forever waiting for db_updated or view_updated events (the later won't come anymore). Like for normal continuous changes feeds we stop if the database is deleted, for _view changes we should stop if the ddoc is deleted (or its "views" attribute is deleted in an update) - this case seems to make sense.
for 3) - I'm ok with that.
for 4) - Non manual tests are good to have :)
I've also noticed after last comment that send_view_changes/2 in couch_changes.erl (https://github.com/benoitc/couchdb/compare/master...couch_view_changes#L4R372) is buffering the full_doc_infos of every document id found in the view when folding it. This seems it can buffer millions (or more) of full_doc_info records, no? If it's really unbounded (as it seems just by looking at the diff), than it's dangerous (even a few thousand full_doc_infos can be too much, if the revision trees have a big depth and/or many branches).,
> improve view filtering in changes
> ---------------------------------
>
> Key: COUCHDB-1398
> URL: https://issues.apache.org/jira/browse/COUCHDB-1398
> Project: CouchDB
> Issue Type: Improvement
> Components: View Server Support
> Affects Versions: 2.0, 1.3
> Reporter: Benoit Chesneau
> Labels: changes, view
> Attachments: 0001-white-spaces.patch, 0002-initial-step-move-the-code-from-couch_httpd_db-to-co.patch, 0003-fix-indent.patch, 0004-This-wrapper-is-useless-somehow-split-the-code-in-a-.patch, 0005-add-view_updated-event.patch, 0006-immprove-view-filter.patch, 0007-useless-info.patch, 0008-whitespaces.patch, 0009-handle-native-filters-in-views.patch
>
>
> Improve the native view filter `_view` support by really using view index. This patches add following features
> - small refactoring: create the couch_httpd_changes modules, to put all the changes http support in its own module instead having it in couch_httpd_db.
> - add the `view_updated` event when a view index is updated : {view_updated, {DbName, IndexName}}
> - start the feed using results in the view index instead of all the db index
> - only react on view index changes.
> For now next changes are still get using the couch_db:changes_since function and passing the map function to the results. It could be improved if we had a by_seq btree in the view index too. Other way may be to skip a number of the documents already processed. Not sure it would be faster. Thoughts ?
> The branch couch_view_changes in my repo contains preliminary support:
> https://github.com/benoitc/couchdb/tree/couch_view_changes
> Diff:
> https://github.com/benoitc/couchdb/compare/master...couch_view_changes
> To use it, use the native filter named _view which take the parameter view=DesignName/Viewname
> eg:
> http://server/db/_changes?feed=continuous&heartbeat=true&filter=_view&view=DesignName/SomeView
> It has also an interresting side effect: on each db updates the view index refresh is triggered so view updates are triggered. Maybe we could introduce an optionnal parameter to not trigger them though?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1398) improve view filtering in changes
Posted by "Benoit Chesneau (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benoit Chesneau updated COUCHDB-1398:
-------------------------------------
Attachment: 0009-handle-native-filters-in-views.patch
0008-whitespaces.patch
handle native filters in replication.
Native filters wasn't handled by the replication. Fix it. While I'm here parse the view argument for the _view filter.
Note: this fix must be ported to 1.2 & last head.
> improve view filtering in changes
> ---------------------------------
>
> Key: COUCHDB-1398
> URL: https://issues.apache.org/jira/browse/COUCHDB-1398
> Project: CouchDB
> Issue Type: Improvement
> Components: View Server Support
> Affects Versions: 2.0, 1.3
> Reporter: Benoit Chesneau
> Labels: changes, view
> Attachments: 0001-white-spaces.patch, 0002-initial-step-move-the-code-from-couch_httpd_db-to-co.patch, 0003-fix-indent.patch, 0004-This-wrapper-is-useless-somehow-split-the-code-in-a-.patch, 0005-add-view_updated-event.patch, 0006-immprove-view-filter.patch, 0007-useless-info.patch, 0008-whitespaces.patch, 0009-handle-native-filters-in-views.patch
>
>
> Improve the native view filter `_view` support by really using view index. This patches add following features
> - small refactoring: create the couch_httpd_changes modules, to put all the changes http support in its own module instead having it in couch_httpd_db.
> - add the `view_updated` event when a view index is updated : {view_updated, {DbName, IndexName}}
> - start the feed using results in the view index instead of all the db index
> - only react on view index changes.
> For now next changes are still get using the couch_db:changes_since function and passing the map function to the results. It could be improved if we had a by_seq btree in the view index too. Other way may be to skip a number of the documents already processed. Not sure it would be faster. Thoughts ?
> The branch couch_view_changes in my repo contains preliminary support:
> https://github.com/benoitc/couchdb/tree/couch_view_changes
> Diff:
> https://github.com/benoitc/couchdb/compare/master...couch_view_changes
> To use it, use the native filter named _view which take the parameter view=DesignName/Viewname
> eg:
> http://server/db/_changes?feed=continuous&heartbeat=true&filter=_view&view=DesignName/SomeView
> It has also an interresting side effect: on each db updates the view index refresh is triggered so view updates are triggered. Maybe we could introduce an optionnal parameter to not trigger them though?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1398) improve view filtering in changes
Posted by "Filipe Manana (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207768#comment-13207768 ]
Filipe Manana commented on COUCHDB-1398:
----------------------------------------
Hi Benoît,
A few comments after quickly looking at the full diff (sorry had no time to test).
1) There seems to be a few white space only unintentional changes that don't eliminate trailing white spaces or
fix the indentation to comply to the overall indentation style. E.g.:
https://github.com/benoitc/couchdb/compare/master...couch_view_changes#L4L357
2) In the make_update_fun:
https://github.com/benoitc/couchdb/compare/master...couch_view_changes#L4R114
I see a few issues there.
First, if the design document is deleted (the view group is shutdown), and the
_changes feed request is of "continuous" type, the client will hang forever as the code doesn't seem to handle this
and will be forever waiting for more 'view_updated' events which will never come (unless a new ddoc with same _id
is created after).
Second, when it receives a db_updated event, it will spawn a process to trigger the view update. If this process dies,
the parent doesn't know it and thinks all is fine. It can die here because for example
couch_httpd_db:couch_doc_open threw {not_found, deleted} (ddoc deleted, related to case 1 mentioned above).
It should stop when the ddoc is deleted or a ddoc update removes its "views" field (which also shutdowns the view group
as soon as no more clients are using it). Possibly when a view group is shutdown, you'll want to emit an event so that
you can react to it. Inside that process you're also calling couch_db:reopen - you should use couch_db:open[_int] here
because that process never opened the database before.
Third, and this might not be an issue (hard to tell) is when for example the ddoc signature changes (like map function was
updated), you keep going as nothing happened. Clients might want to stop receiving changes if the view definitions of
a ddoc change - this might be very subjective or application dependent. I don't have a strong opinion here.
As of COUCHDB-1309, when a ddoc is updated such that the view group signature changes, it's old view group will
shutdown when no more clients are using it. Perhaps when it's shutdown, emitting an event with the DDocId and Signature
will help dealing with the cases listed above.
First case where a ddoc is deleted, should probably have a test to verify the clients isn't kept blocked forever
after the ddoc is deleted (for continuous feed types).
3) You're reusing couch_db_update_notifier for emitting the view_updated events. Probably the couch_mrview application
should have it's own separate event manager. I don't see this as a big issue or a blocker, can be addressed later.
4) For the replication, some tests would be very welcome and good to have.
Good work and thanks.
> improve view filtering in changes
> ---------------------------------
>
> Key: COUCHDB-1398
> URL: https://issues.apache.org/jira/browse/COUCHDB-1398
> Project: CouchDB
> Issue Type: Improvement
> Components: View Server Support
> Affects Versions: 2.0, 1.3
> Reporter: Benoit Chesneau
> Labels: changes, view
> Attachments: 0001-white-spaces.patch, 0002-initial-step-move-the-code-from-couch_httpd_db-to-co.patch, 0003-fix-indent.patch, 0004-This-wrapper-is-useless-somehow-split-the-code-in-a-.patch, 0005-add-view_updated-event.patch, 0006-immprove-view-filter.patch, 0007-useless-info.patch, 0008-whitespaces.patch, 0009-handle-native-filters-in-views.patch
>
>
> Improve the native view filter `_view` support by really using view index. This patches add following features
> - small refactoring: create the couch_httpd_changes modules, to put all the changes http support in its own module instead having it in couch_httpd_db.
> - add the `view_updated` event when a view index is updated : {view_updated, {DbName, IndexName}}
> - start the feed using results in the view index instead of all the db index
> - only react on view index changes.
> For now next changes are still get using the couch_db:changes_since function and passing the map function to the results. It could be improved if we had a by_seq btree in the view index too. Other way may be to skip a number of the documents already processed. Not sure it would be faster. Thoughts ?
> The branch couch_view_changes in my repo contains preliminary support:
> https://github.com/benoitc/couchdb/tree/couch_view_changes
> Diff:
> https://github.com/benoitc/couchdb/compare/master...couch_view_changes
> To use it, use the native filter named _view which take the parameter view=DesignName/Viewname
> eg:
> http://server/db/_changes?feed=continuous&heartbeat=true&filter=_view&view=DesignName/SomeView
> It has also an interresting side effect: on each db updates the view index refresh is triggered so view updates are triggered. Maybe we could introduce an optionnal parameter to not trigger them though?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira