You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Chris Stockton (JIRA)" <ji...@apache.org> on 2011/08/23 02:15:29 UTC
[jira] [Created] (COUCHDB-1257) Reduction detection in
share/server/views.js runReduce to primitive for some use cases and overall
reduce architecture
Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
----------------------------------------------------------------------------------------------------------------------
Key: COUCHDB-1257
URL: https://issues.apache.org/jira/browse/COUCHDB-1257
Project: CouchDB
Issue Type: Bug
Components: JavaScript View Server
Affects Versions: 1.1
Environment: Architecture independent
Reporter: Chris Stockton
Priority: Minor
There is two parts to this report, one is my question I have in regards to the protocol in which couchdb actually talks to the view server (server/main.js), the second part as discussed here [1] in short is the growth detection [2] done in share/server/views.js.
First issue:
While troubleshooting and tracking down this issue, I saw that when running a reduce function for every doc called, a separate read line is delegated a reset call. I do not know the internals of couchdb but it seems like it could be a performance improvement if on a rereduce call, as it appears in the server Loop function every time reduce or rereduce is called the state is first reset with "reset", to make a minor re-factor and reset the state within the Views dispatcher itself. This saves some likely blocking io back and forth.
Reduction detection:
The portion of this issue that actually has affected us is the detection done, I think that it should be changed to require a sampling of data, the easiest fix is to simply change (the very arbitrary) [2] into a large length, such as * 10 or something, but the upper bounds will always be reached for some data set. A new algorithm could take in consideration the total reduce calls thus far, as well as the initial size of the very first reduce call, to see if a large aggregation of some sort is being created.
Suggested change:
Change the signature of rereduce call to main.js to include the total count of rereduce calls thus far, or give some kind of handle where this logic could be reliably kept track of inside of main.js, as it stands now your only hope would be to use the reduce function itself as a lookup, which is not safe under all edge cases. With this information you can keep the initial rereduce calls result size, and the total number of calls, to make sure that aside from initial aggregation or structure initialization the structure is remaining constant.
There would be several other routes to go to fix this, but I think it is valid to be considered. I could volunteer my time to write a patch if a consensus could be reached on behavior, if not denied all together.
-Chris
[1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
[2] if (State.query_config && State.query_config.reduce_limit &&
[2] reduce_length > 200 && ((reduce_length * 2) > State.line_length)) {
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1257) Reduction detection in
share/server/views.js runReduce to primitive for some use cases and overall
reduce architecture
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089210#comment-13089210 ]
Adam Kocoloski commented on COUCHDB-1257:
-----------------------------------------
Hi Chris, thanks for this ticket. I agree with the subject line, I'll have to take a closer look at the suggested changes.
> Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: COUCHDB-1257
> URL: https://issues.apache.org/jira/browse/COUCHDB-1257
> Project: CouchDB
> Issue Type: Bug
> Components: JavaScript View Server
> Affects Versions: 1.1
> Environment: Architecture independent
> Reporter: Chris Stockton
> Priority: Minor
> Labels: javascript, patch
>
> There is two parts to this report, one is my question I have in regards to the protocol in which couchdb actually talks to the view server (server/main.js), the second part as discussed here [1] in short is the growth detection [2] done in share/server/views.js.
> First issue:
> While troubleshooting and tracking down this issue, I saw that when running a reduce function for every doc called, a separate read line is delegated a reset call. I do not know the internals of couchdb but it seems like it could be a performance improvement if on a rereduce call, as it appears in the server Loop function every time reduce or rereduce is called the state is first reset with "reset", to make a minor re-factor and reset the state within the Views dispatcher itself. This saves some likely blocking io back and forth.
> Reduction detection:
> The portion of this issue that actually has affected us is the detection done, I think that it should be changed to require a sampling of data, the easiest fix is to simply change (the very arbitrary) [2] into a large length, such as * 10 or something, but the upper bounds will always be reached for some data set. A new algorithm could take in consideration the total reduce calls thus far, as well as the initial size of the very first reduce call, to see if a large aggregation of some sort is being created.
> Suggested change:
> Change the signature of rereduce call to main.js to include the total count of rereduce calls thus far, or give some kind of handle where this logic could be reliably kept track of inside of main.js, as it stands now your only hope would be to use the reduce function itself as a lookup, which is not safe under all edge cases. With this information you can keep the initial rereduce calls result size, and the total number of calls, to make sure that aside from initial aggregation or structure initialization the structure is remaining constant.
> There would be several other routes to go to fix this, but I think it is valid to be considered. I could volunteer my time to write a patch if a consensus could be reached on behavior, if not denied all together.
> -Chris
> [1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
> [2] if (State.query_config && State.query_config.reduce_limit &&
> [2] reduce_length > 200 && ((reduce_length * 2) > State.line_length)) {
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1257) Reduction detection in
share/server/views.js runReduce to primitive for some use cases and overall
reduce architecture
Posted by "Paul Joseph Davis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090551#comment-13090551 ]
Paul Joseph Davis commented on COUCHDB-1257:
--------------------------------------------
Totally just realized this is the same issue as the mailing list. Either way, yeah...
> Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: COUCHDB-1257
> URL: https://issues.apache.org/jira/browse/COUCHDB-1257
> Project: CouchDB
> Issue Type: Bug
> Components: JavaScript View Server
> Affects Versions: 1.1
> Environment: Architecture independent
> Reporter: Chris Stockton
> Priority: Minor
> Labels: javascript, patch
>
> There is two parts to this report, one is my question I have in regards to the protocol in which couchdb actually talks to the view server (server/main.js), the second part as discussed here [1] in short is the growth detection [2] done in share/server/views.js.
> First issue:
> While troubleshooting and tracking down this issue, I saw that when running a reduce function for every doc called, a separate read line is delegated a reset call. I do not know the internals of couchdb but it seems like it could be a performance improvement if on a rereduce call, as it appears in the server Loop function every time reduce or rereduce is called the state is first reset with "reset", to make a minor re-factor and reset the state within the Views dispatcher itself. This saves some likely blocking io back and forth.
> Reduction detection:
> The portion of this issue that actually has affected us is the detection done, I think that it should be changed to require a sampling of data, the easiest fix is to simply change (the very arbitrary) [2] into a large length, such as * 10 or something, but the upper bounds will always be reached for some data set. A new algorithm could take in consideration the total reduce calls thus far, as well as the initial size of the very first reduce call, to see if a large aggregation of some sort is being created.
> Suggested change:
> Change the signature of rereduce call to main.js to include the total count of rereduce calls thus far, or give some kind of handle where this logic could be reliably kept track of inside of main.js, as it stands now your only hope would be to use the reduce function itself as a lookup, which is not safe under all edge cases. With this information you can keep the initial rereduce calls result size, and the total number of calls, to make sure that aside from initial aggregation or structure initialization the structure is remaining constant.
> There would be several other routes to go to fix this, but I think it is valid to be considered. I could volunteer my time to write a patch if a consensus could be reached on behavior, if not denied all together.
> -Chris
> [1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
> [2] if (State.query_config && State.query_config.reduce_limit &&
> [2] reduce_length > 200 && ((reduce_length * 2) > State.line_length)) {
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1257) Reduction detection in
share/server/views.js runReduce to primitive for some use cases and overall
reduce architecture
Posted by "Paul Joseph Davis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090548#comment-13090548 ]
Paul Joseph Davis commented on COUCHDB-1257:
--------------------------------------------
We just fixed this in trunk. It's a small patch to fix the issue if you're interested. You only need the couch_os_file.erl and couch_query_servers.erl diffs to make it work.
https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9
> Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: COUCHDB-1257
> URL: https://issues.apache.org/jira/browse/COUCHDB-1257
> Project: CouchDB
> Issue Type: Bug
> Components: JavaScript View Server
> Affects Versions: 1.1
> Environment: Architecture independent
> Reporter: Chris Stockton
> Priority: Minor
> Labels: javascript, patch
>
> There is two parts to this report, one is my question I have in regards to the protocol in which couchdb actually talks to the view server (server/main.js), the second part as discussed here [1] in short is the growth detection [2] done in share/server/views.js.
> First issue:
> While troubleshooting and tracking down this issue, I saw that when running a reduce function for every doc called, a separate read line is delegated a reset call. I do not know the internals of couchdb but it seems like it could be a performance improvement if on a rereduce call, as it appears in the server Loop function every time reduce or rereduce is called the state is first reset with "reset", to make a minor re-factor and reset the state within the Views dispatcher itself. This saves some likely blocking io back and forth.
> Reduction detection:
> The portion of this issue that actually has affected us is the detection done, I think that it should be changed to require a sampling of data, the easiest fix is to simply change (the very arbitrary) [2] into a large length, such as * 10 or something, but the upper bounds will always be reached for some data set. A new algorithm could take in consideration the total reduce calls thus far, as well as the initial size of the very first reduce call, to see if a large aggregation of some sort is being created.
> Suggested change:
> Change the signature of rereduce call to main.js to include the total count of rereduce calls thus far, or give some kind of handle where this logic could be reliably kept track of inside of main.js, as it stands now your only hope would be to use the reduce function itself as a lookup, which is not safe under all edge cases. With this information you can keep the initial rereduce calls result size, and the total number of calls, to make sure that aside from initial aggregation or structure initialization the structure is remaining constant.
> There would be several other routes to go to fix this, but I think it is valid to be considered. I could volunteer my time to write a patch if a consensus could be reached on behavior, if not denied all together.
> -Chris
> [1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
> [2] if (State.query_config && State.query_config.reduce_limit &&
> [2] reduce_length > 200 && ((reduce_length * 2) > State.line_length)) {
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1257) Reduction detection in
share/server/views.js runReduce to primitive for some use cases and overall
reduce architecture
Posted by "Chris Stockton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090484#comment-13090484 ]
Chris Stockton commented on COUCHDB-1257:
-----------------------------------------
Adam it appears this error is a bit more serious then original understood. While this has been happening we were having a similar error with views no longer returning results and just timing out. It seems each reduce over flow error results in a couchjs process remaining open forever.
[root@p3nldbnowcdb01 ~]# ps auxww|grep -i main.js|wc -l
14
<induce error 10 times>
[root@p3nldbnowcdb01 ~]# ps auxww|grep -i main.js|wc -l
24
This ultimately leads to couch becoming unstable.
> Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: COUCHDB-1257
> URL: https://issues.apache.org/jira/browse/COUCHDB-1257
> Project: CouchDB
> Issue Type: Bug
> Components: JavaScript View Server
> Affects Versions: 1.1
> Environment: Architecture independent
> Reporter: Chris Stockton
> Priority: Minor
> Labels: javascript, patch
>
> There is two parts to this report, one is my question I have in regards to the protocol in which couchdb actually talks to the view server (server/main.js), the second part as discussed here [1] in short is the growth detection [2] done in share/server/views.js.
> First issue:
> While troubleshooting and tracking down this issue, I saw that when running a reduce function for every doc called, a separate read line is delegated a reset call. I do not know the internals of couchdb but it seems like it could be a performance improvement if on a rereduce call, as it appears in the server Loop function every time reduce or rereduce is called the state is first reset with "reset", to make a minor re-factor and reset the state within the Views dispatcher itself. This saves some likely blocking io back and forth.
> Reduction detection:
> The portion of this issue that actually has affected us is the detection done, I think that it should be changed to require a sampling of data, the easiest fix is to simply change (the very arbitrary) [2] into a large length, such as * 10 or something, but the upper bounds will always be reached for some data set. A new algorithm could take in consideration the total reduce calls thus far, as well as the initial size of the very first reduce call, to see if a large aggregation of some sort is being created.
> Suggested change:
> Change the signature of rereduce call to main.js to include the total count of rereduce calls thus far, or give some kind of handle where this logic could be reliably kept track of inside of main.js, as it stands now your only hope would be to use the reduce function itself as a lookup, which is not safe under all edge cases. With this information you can keep the initial rereduce calls result size, and the total number of calls, to make sure that aside from initial aggregation or structure initialization the structure is remaining constant.
> There would be several other routes to go to fix this, but I think it is valid to be considered. I could volunteer my time to write a patch if a consensus could be reached on behavior, if not denied all together.
> -Chris
> [1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
> [2] if (State.query_config && State.query_config.reduce_limit &&
> [2] reduce_length > 200 && ((reduce_length * 2) > State.line_length)) {
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira