You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Glenn Rempe (JIRA)" <ji...@apache.org> on 2009/10/17 00:49:31 UTC
[jira] Created: (COUCHDB-532) Have requests for views that use
'stale=ok' return whats in the index immediately, but still kickoff
background indexing process.
Have requests for views that use 'stale=ok' return whats in the index immediately, but still kickoff background indexing process.
---------------------------------------------------------------------------------------------------------------------------------
Key: COUCHDB-532
URL: https://issues.apache.org/jira/browse/COUCHDB-532
Project: CouchDB
Issue Type: Improvement
Components: Database Core
Environment: All
Reporter: Glenn Rempe
Currently a common use case for querying indexes from external applications is to pass in the optional parameter 'stale=ok' which instructs the view to return what is has in its index immediately without waiting for the index to first catch up.
This is a *very* important feature as it allows applications to eliminate the poor user experience of potential HTTP client timeouts or long'ish waits for a query to return while the index is updated.
However, applications that make extensive (or in my case exlusive) use of this feature are now essentially *required* to build some kind of script which will do the indexing in the background since no user requests will *ever* initiate indexing. This would be for example a cron script that makes an HTTP call to the view every N hours. This becomes an operational issue as you now have to write, maintain and schedule an external script which calls each view in each of your DBs periodically.
I propose to the team, after some discussion with Paul Joseph Davis, that view requests with the 'stale=ok' param continue to return results in the as-is index to users immediately, but *also* kick off the background indexing process. This would have the effect of removing the need for any external maintenance scripts (aside from DB/View compaction which is a related but separate issue).
In some use cases it might be true that indexing *must* occur outside of 'normal business hours' due to processing load. If this is a requirement the behavior of view indexing on 'stale' requests could be configurable (e.g. as a boolean to always/never do this additional background indexing, or force BG indexing every N requests/minutes using 'stale').
Cheers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-532) Have requests for views that use
'stale=ok' return whats in the index immediately, but still kickoff
background indexing process.
Posted by "Jan Lehnardt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766904#action_12766904 ]
Jan Lehnardt commented on COUCHDB-532:
--------------------------------------
I think we have plans for a facility that fixes the underlying issue here without the proposed way of solving it. Adam and I are dreaming up a "task manager" sorta like cron, that lets you define arbitrary actions to be run on events, they could be time related, based on stats counters or just continuous. for example continuous replication that is supposed to survive a server restart is one of these things. or scheduling compaction to run each night at 2am. another part is keeping views up to date asynchronously, so clients can use stale=ok all the time. I'm pretty sure we a ticket for this, but I couldn't find it just now.
> Have requests for views that use 'stale=ok' return whats in the index immediately, but still kickoff background indexing process.
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: COUCHDB-532
> URL: https://issues.apache.org/jira/browse/COUCHDB-532
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Environment: All
> Reporter: Glenn Rempe
>
> Currently a common use case for querying indexes from external applications is to pass in the optional parameter 'stale=ok' which instructs the view to return what is has in its index immediately without waiting for the index to first catch up.
> This is a *very* important feature as it allows applications to eliminate the poor user experience of potential HTTP client timeouts or long'ish waits for a query to return while the index is updated.
> However, applications that make extensive (or in my case exlusive) use of this feature are now essentially *required* to build some kind of script which will do the indexing in the background since no user requests will *ever* initiate indexing. This would be for example a cron script that makes an HTTP call to the view every N hours. This becomes an operational issue as you now have to write, maintain and schedule an external script which calls each view in each of your DBs periodically.
> I propose to the team, after some discussion with Paul Joseph Davis, that view requests with the 'stale=ok' param continue to return results in the as-is index to users immediately, but *also* kick off the background indexing process. This would have the effect of removing the need for any external maintenance scripts (aside from DB/View compaction which is a related but separate issue).
> In some use cases it might be true that indexing *must* occur outside of 'normal business hours' due to processing load. If this is a requirement the behavior of view indexing on 'stale' requests could be configurable (e.g. as a boolean to always/never do this additional background indexing, or force BG indexing every N requests/minutes using 'stale').
> Cheers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-532) Have requests for views that use
'stale=ok' return whats in the index immediately, but still kickoff
background indexing process.
Posted by "Jan Lehnardt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766941#action_12766941 ]
Jan Lehnardt commented on COUCHDB-532:
--------------------------------------
I want this for 0.11 ;)
I don't like adding stopgap cruft to the already complex view engine code.
> Have requests for views that use 'stale=ok' return whats in the index immediately, but still kickoff background indexing process.
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: COUCHDB-532
> URL: https://issues.apache.org/jira/browse/COUCHDB-532
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Environment: All
> Reporter: Glenn Rempe
>
> Currently a common use case for querying indexes from external applications is to pass in the optional parameter 'stale=ok' which instructs the view to return what is has in its index immediately without waiting for the index to first catch up.
> This is a *very* important feature as it allows applications to eliminate the poor user experience of potential HTTP client timeouts or long'ish waits for a query to return while the index is updated.
> However, applications that make extensive (or in my case exlusive) use of this feature are now essentially *required* to build some kind of script which will do the indexing in the background since no user requests will *ever* initiate indexing. This would be for example a cron script that makes an HTTP call to the view every N hours. This becomes an operational issue as you now have to write, maintain and schedule an external script which calls each view in each of your DBs periodically.
> I propose to the team, after some discussion with Paul Joseph Davis, that view requests with the 'stale=ok' param continue to return results in the as-is index to users immediately, but *also* kick off the background indexing process. This would have the effect of removing the need for any external maintenance scripts (aside from DB/View compaction which is a related but separate issue).
> In some use cases it might be true that indexing *must* occur outside of 'normal business hours' due to processing load. If this is a requirement the behavior of view indexing on 'stale' requests could be configurable (e.g. as a boolean to always/never do this additional background indexing, or force BG indexing every N requests/minutes using 'stale').
> Cheers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-532) Have requests for views that use
'stale=ok' return whats in the index immediately, but still kickoff
background indexing process.
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767151#action_12767151 ]
Adam Kocoloski commented on COUCHDB-532:
----------------------------------------
I can certainly add the automatic async view updating in time for 0.11 if no one else does.
> Have requests for views that use 'stale=ok' return whats in the index immediately, but still kickoff background indexing process.
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: COUCHDB-532
> URL: https://issues.apache.org/jira/browse/COUCHDB-532
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Environment: All
> Reporter: Glenn Rempe
>
> Currently a common use case for querying indexes from external applications is to pass in the optional parameter 'stale=ok' which instructs the view to return what is has in its index immediately without waiting for the index to first catch up.
> This is a *very* important feature as it allows applications to eliminate the poor user experience of potential HTTP client timeouts or long'ish waits for a query to return while the index is updated.
> However, applications that make extensive (or in my case exlusive) use of this feature are now essentially *required* to build some kind of script which will do the indexing in the background since no user requests will *ever* initiate indexing. This would be for example a cron script that makes an HTTP call to the view every N hours. This becomes an operational issue as you now have to write, maintain and schedule an external script which calls each view in each of your DBs periodically.
> I propose to the team, after some discussion with Paul Joseph Davis, that view requests with the 'stale=ok' param continue to return results in the as-is index to users immediately, but *also* kick off the background indexing process. This would have the effect of removing the need for any external maintenance scripts (aside from DB/View compaction which is a related but separate issue).
> In some use cases it might be true that indexing *must* occur outside of 'normal business hours' due to processing load. If this is a requirement the behavior of view indexing on 'stale' requests could be configurable (e.g. as a boolean to always/never do this additional background indexing, or force BG indexing every N requests/minutes using 'stale').
> Cheers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-532) Have requests for views that use
'stale=ok' return whats in the index immediately, but still kickoff
background indexing process.
Posted by "Glenn Rempe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766938#action_12766938 ]
Glenn Rempe commented on COUCHDB-532:
-------------------------------------
Jan, This would, in my opinion, be an excellent new facility to have. Based on your description it would indeed solve the issue at hand nicely if it were time and event based. Allowing for cron-like control over all of the major aspects of CouchDB would be powerful.
Would you say this is a long term feature plan that is early in its design, or one that is close to reaching fruition? If its a longer term feature plan, perhaps the enhancement to do background indexing on stale=ok could be implemented as a stop-gap measure in the interim if its easy to add (and just as easy to remove later)? Speaking for myself I could use this feature yesterday. :-)
> Have requests for views that use 'stale=ok' return whats in the index immediately, but still kickoff background indexing process.
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: COUCHDB-532
> URL: https://issues.apache.org/jira/browse/COUCHDB-532
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Environment: All
> Reporter: Glenn Rempe
>
> Currently a common use case for querying indexes from external applications is to pass in the optional parameter 'stale=ok' which instructs the view to return what is has in its index immediately without waiting for the index to first catch up.
> This is a *very* important feature as it allows applications to eliminate the poor user experience of potential HTTP client timeouts or long'ish waits for a query to return while the index is updated.
> However, applications that make extensive (or in my case exlusive) use of this feature are now essentially *required* to build some kind of script which will do the indexing in the background since no user requests will *ever* initiate indexing. This would be for example a cron script that makes an HTTP call to the view every N hours. This becomes an operational issue as you now have to write, maintain and schedule an external script which calls each view in each of your DBs periodically.
> I propose to the team, after some discussion with Paul Joseph Davis, that view requests with the 'stale=ok' param continue to return results in the as-is index to users immediately, but *also* kick off the background indexing process. This would have the effect of removing the need for any external maintenance scripts (aside from DB/View compaction which is a related but separate issue).
> In some use cases it might be true that indexing *must* occur outside of 'normal business hours' due to processing load. If this is a requirement the behavior of view indexing on 'stale' requests could be configurable (e.g. as a boolean to always/never do this additional background indexing, or force BG indexing every N requests/minutes using 'stale').
> Cheers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (COUCHDB-532) Have requests for views that use
'stale=ok' return whats in the index immediately, but still kickoff
background indexing process.
Posted by "Paul Joseph Davis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Joseph Davis resolved COUCHDB-532.
---------------------------------------
Resolution: Fixed
Fixed by fdmanana's update_after patch.
> Have requests for views that use 'stale=ok' return whats in the index immediately, but still kickoff background indexing process.
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: COUCHDB-532
> URL: https://issues.apache.org/jira/browse/COUCHDB-532
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Environment: All
> Reporter: Glenn Rempe
>
> Currently a common use case for querying indexes from external applications is to pass in the optional parameter 'stale=ok' which instructs the view to return what is has in its index immediately without waiting for the index to first catch up.
> This is a *very* important feature as it allows applications to eliminate the poor user experience of potential HTTP client timeouts or long'ish waits for a query to return while the index is updated.
> However, applications that make extensive (or in my case exlusive) use of this feature are now essentially *required* to build some kind of script which will do the indexing in the background since no user requests will *ever* initiate indexing. This would be for example a cron script that makes an HTTP call to the view every N hours. This becomes an operational issue as you now have to write, maintain and schedule an external script which calls each view in each of your DBs periodically.
> I propose to the team, after some discussion with Paul Joseph Davis, that view requests with the 'stale=ok' param continue to return results in the as-is index to users immediately, but *also* kick off the background indexing process. This would have the effect of removing the need for any external maintenance scripts (aside from DB/View compaction which is a related but separate issue).
> In some use cases it might be true that indexing *must* occur outside of 'normal business hours' due to processing load. If this is a requirement the behavior of view indexing on 'stale' requests could be configurable (e.g. as a boolean to always/never do this additional background indexing, or force BG indexing every N requests/minutes using 'stale').
> Cheers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.