You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Sami Sierla <sa...@poplatek.fi> on 2013/03/08 15:18:01 UTC
Very slow queries to view in design doc
Hi,
We are having odd issue with CouchDB 1.2.0 view queries - queries to one particular view are very slow (tens of seconds with limit=1) but queries to other views in the same design doc perform as they should. Issue goes away if we restart CouchDB but comes back after few hours. Views are periodically refreshed so there are no indexing blocking the query. The setup we have consists of about ten CouchDB databases on active db server (RHEL 5) and second standby server with continuous CouchDB pull replication. Same view query works fine on the standby server that has no other processes running.
Any ideas what could cause this?
Best,
Sami Sierla
Re: Very slow queries to view in design doc
Posted by Sami Sierla <sa...@poplatek.fi>.
Hi,
In the end we just chose to drop reduce function (which was a bit overkill) completely, fetch all view entries (there are usually less than 100 entries / query) and do summary calculation on Java side.
-Sami
On Mar 12, 2013, at 6:04 PM, Robert Newson <rn...@apache.org> wrote:
> Your reduce function does not reduce.
>
> I recommend looping in your map function to emit a row for each of the
> attributes you wish to sum and then use the built-in "_sum" reduce
> function. I think you'll find it not only much faster, but it won't
> break.
>
> B.
>
> On 12 March 2013 04:50, Sami Sierla <sa...@poplatek.fi> wrote:
>> Hi,
>>
>> Issue happened again today. I was able to check file descriptors and established tcp connections to CouchDB. file descriptors in use were about 80 so they are not hitting any limits. Number of connections to couch before and after reboot were about 130.
>>
>> Reduce function counts sums from document attributes - there are about one million documents in view.
>>
>> Here's the raw json of map / reduce function:
>>
>> "Summary": {
>> "map": "function(doc) { if (doc.objectType == \"Document\" && (doc.severity !== 4)) { emit(doc.alertTime, {\"feeds\": doc.feeds, \"severity\": doc.severity}); } }",
>> "reduce": "function(keys, values, rereduce) {\n var i, j, k, v, res, feeds;\n res = {};\n if (!rereduce) {\n res[\"*,*\"] = values.length;\n for (i = 0; i < values.length; i++) {\n k = \"*,\"+values[i].severity;\n res[k] = (res[k] || 0) + 1;\n feeds = values[i].feeds;\n for (j = 0; j < feeds.length; j++) {\n k = feeds[j]+\",*\";\n res[k] = (res[k] || 0) + 1;\n k = feeds[j]+\",\"+values[i].severity;\n res[k] = (res[k] || 0) + 1;\n }\n }\n } else {\n for (i = 0; i < values.length; i++) {\n v = values[i];\n for (k in v) {\n res[k] = (res[k] || 0) + v[k];\n }\n }\n }\n return res;\n}\n"
>> }
>>
>> View entries are like this:
>> {"id":"ac:431590","key":"2010-11-01T00:30:02.000+0000","value":{"feeds":["BNK"],"severity":2}}
>>
>> Again, this works just fine after reboot (even with reduce) but seems to slow over time (about 24-48 hours) until it starts to timeout.
>>
>> -Sami
>>
>> On Mar 11, 2013, at 11:20 PM, Robert Newson <rn...@apache.org> wrote:
>>
>>> Can you access /_log on the server? might have information.
>>>
>>> Does the custom reduce function actually reduce (that is, return a
>>> much smaller value than the input 'values' array)?
>>>
>>> B.
>>>
>>> On 11 March 2013 03:27, Sami Sierla <sa...@poplatek.fi> wrote:
>>>> Hi,
>>>>
>>>> There is custom reduce function in this view but result is same even if reduce=false. Could this have something to do with running out of file descriptors (I unfortunately don't have sufficient user rights to check this on server). Issue began after we added two databases (unrelated to this view) to CouchDB (both have continuous replication to standby server).
>>>>
>>>> Best,
>>>> Sami
>>>>
>>>> On Mar 8, 2013, at 4:20 PM, Robert Newson <rn...@apache.org> wrote:
>>>>
>>>>> Do you have a custom reduce function? Are the keys/values in that view
>>>>> very large?
>>>>>
>>>>> On 8 March 2013 08:18, Sami Sierla <sa...@poplatek.fi> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> We are having odd issue with CouchDB 1.2.0 view queries - queries to one particular view are very slow (tens of seconds with limit=1) but queries to other views in the same design doc perform as they should. Issue goes away if we restart CouchDB but comes back after few hours. Views are periodically refreshed so there are no indexing blocking the query. The setup we have consists of about ten CouchDB databases on active db server (RHEL 5) and second standby server with continuous CouchDB pull replication. Same view query works fine on the standby server that has no other processes running.
>>>>>>
>>>>>> Any ideas what could cause this?
>>>>>>
>>>>>> Best,
>>>>>> Sami Sierla
>>>>
>>
Re: Very slow queries to view in design doc
Posted by Robert Newson <rn...@apache.org>.
Your reduce function does not reduce.
I recommend looping in your map function to emit a row for each of the
attributes you wish to sum and then use the built-in "_sum" reduce
function. I think you'll find it not only much faster, but it won't
break.
B.
On 12 March 2013 04:50, Sami Sierla <sa...@poplatek.fi> wrote:
> Hi,
>
> Issue happened again today. I was able to check file descriptors and established tcp connections to CouchDB. file descriptors in use were about 80 so they are not hitting any limits. Number of connections to couch before and after reboot were about 130.
>
> Reduce function counts sums from document attributes - there are about one million documents in view.
>
> Here's the raw json of map / reduce function:
>
> "Summary": {
> "map": "function(doc) { if (doc.objectType == \"Document\" && (doc.severity !== 4)) { emit(doc.alertTime, {\"feeds\": doc.feeds, \"severity\": doc.severity}); } }",
> "reduce": "function(keys, values, rereduce) {\n var i, j, k, v, res, feeds;\n res = {};\n if (!rereduce) {\n res[\"*,*\"] = values.length;\n for (i = 0; i < values.length; i++) {\n k = \"*,\"+values[i].severity;\n res[k] = (res[k] || 0) + 1;\n feeds = values[i].feeds;\n for (j = 0; j < feeds.length; j++) {\n k = feeds[j]+\",*\";\n res[k] = (res[k] || 0) + 1;\n k = feeds[j]+\",\"+values[i].severity;\n res[k] = (res[k] || 0) + 1;\n }\n }\n } else {\n for (i = 0; i < values.length; i++) {\n v = values[i];\n for (k in v) {\n res[k] = (res[k] || 0) + v[k];\n }\n }\n }\n return res;\n}\n"
> }
>
> View entries are like this:
> {"id":"ac:431590","key":"2010-11-01T00:30:02.000+0000","value":{"feeds":["BNK"],"severity":2}}
>
> Again, this works just fine after reboot (even with reduce) but seems to slow over time (about 24-48 hours) until it starts to timeout.
>
> -Sami
>
> On Mar 11, 2013, at 11:20 PM, Robert Newson <rn...@apache.org> wrote:
>
>> Can you access /_log on the server? might have information.
>>
>> Does the custom reduce function actually reduce (that is, return a
>> much smaller value than the input 'values' array)?
>>
>> B.
>>
>> On 11 March 2013 03:27, Sami Sierla <sa...@poplatek.fi> wrote:
>>> Hi,
>>>
>>> There is custom reduce function in this view but result is same even if reduce=false. Could this have something to do with running out of file descriptors (I unfortunately don't have sufficient user rights to check this on server). Issue began after we added two databases (unrelated to this view) to CouchDB (both have continuous replication to standby server).
>>>
>>> Best,
>>> Sami
>>>
>>> On Mar 8, 2013, at 4:20 PM, Robert Newson <rn...@apache.org> wrote:
>>>
>>>> Do you have a custom reduce function? Are the keys/values in that view
>>>> very large?
>>>>
>>>> On 8 March 2013 08:18, Sami Sierla <sa...@poplatek.fi> wrote:
>>>>> Hi,
>>>>>
>>>>> We are having odd issue with CouchDB 1.2.0 view queries - queries to one particular view are very slow (tens of seconds with limit=1) but queries to other views in the same design doc perform as they should. Issue goes away if we restart CouchDB but comes back after few hours. Views are periodically refreshed so there are no indexing blocking the query. The setup we have consists of about ten CouchDB databases on active db server (RHEL 5) and second standby server with continuous CouchDB pull replication. Same view query works fine on the standby server that has no other processes running.
>>>>>
>>>>> Any ideas what could cause this?
>>>>>
>>>>> Best,
>>>>> Sami Sierla
>>>
>
Re: Very slow queries to view in design doc
Posted by Jeff Charette <io...@yahoo.com>.
Is there a timeout message?
Jeff Charette | Principal
We Are Charette
web / identity / packaging
m 415.298.2707
w wearecharette.com
e jeffrey@wearecharette.com
On Mar 12, 2013, at 5:50 AM, Sami Sierla <sa...@poplatek.fi> wrote:
> Hi,
>
> Issue happened again today. I was able to check file descriptors and established tcp connections to CouchDB. file descriptors in use were about 80 so they are not hitting any limits. Number of connections to couch before and after reboot were about 130.
>
> Reduce function counts sums from document attributes - there are about one million documents in view.
>
> Here's the raw json of map / reduce function:
>
> "Summary": {
> "map": "function(doc) { if (doc.objectType == \"Document\" && (doc.severity !== 4)) { emit(doc.alertTime, {\"feeds\": doc.feeds, \"severity\": doc.severity}); } }",
> "reduce": "function(keys, values, rereduce) {\n var i, j, k, v, res, feeds;\n res = {};\n if (!rereduce) {\n res[\"*,*\"] = values.length;\n for (i = 0; i < values.length; i++) {\n k = \"*,\"+values[i].severity;\n res[k] = (res[k] || 0) + 1;\n feeds = values[i].feeds;\n for (j = 0; j < feeds.length; j++) {\n k = feeds[j]+\",*\";\n res[k] = (res[k] || 0) + 1;\n k = feeds[j]+\",\"+values[i].severity;\n res[k] = (res[k] || 0) + 1;\n }\n }\n } else {\n for (i = 0; i < values.length; i++) {\n v = values[i];\n for (k in v) {\n res[k] = (res[k] || 0) + v[k];\n }\n }\n }\n return res;\n}\n"
> }
>
> View entries are like this:
> {"id":"ac:431590","key":"2010-11-01T00:30:02.000+0000","value":{"feeds":["BNK"],"severity":2}}
>
> Again, this works just fine after reboot (even with reduce) but seems to slow over time (about 24-48 hours) until it starts to timeout.
>
> -Sami
>
> On Mar 11, 2013, at 11:20 PM, Robert Newson <rn...@apache.org> wrote:
>
>> Can you access /_log on the server? might have information.
>>
>> Does the custom reduce function actually reduce (that is, return a
>> much smaller value than the input 'values' array)?
>>
>> B.
>>
>> On 11 March 2013 03:27, Sami Sierla <sa...@poplatek.fi> wrote:
>>> Hi,
>>>
>>> There is custom reduce function in this view but result is same even if reduce=false. Could this have something to do with running out of file descriptors (I unfortunately don't have sufficient user rights to check this on server). Issue began after we added two databases (unrelated to this view) to CouchDB (both have continuous replication to standby server).
>>>
>>> Best,
>>> Sami
>>>
>>> On Mar 8, 2013, at 4:20 PM, Robert Newson <rn...@apache.org> wrote:
>>>
>>>> Do you have a custom reduce function? Are the keys/values in that view
>>>> very large?
>>>>
>>>> On 8 March 2013 08:18, Sami Sierla <sa...@poplatek.fi> wrote:
>>>>> Hi,
>>>>>
>>>>> We are having odd issue with CouchDB 1.2.0 view queries - queries to one particular view are very slow (tens of seconds with limit=1) but queries to other views in the same design doc perform as they should. Issue goes away if we restart CouchDB but comes back after few hours. Views are periodically refreshed so there are no indexing blocking the query. The setup we have consists of about ten CouchDB databases on active db server (RHEL 5) and second standby server with continuous CouchDB pull replication. Same view query works fine on the standby server that has no other processes running.
>>>>>
>>>>> Any ideas what could cause this?
>>>>>
>>>>> Best,
>>>>> Sami Sierla
>>>
>
Re: Very slow queries to view in design doc
Posted by Sami Sierla <sa...@poplatek.fi>.
Hi,
Issue happened again today. I was able to check file descriptors and established tcp connections to CouchDB. file descriptors in use were about 80 so they are not hitting any limits. Number of connections to couch before and after reboot were about 130.
Reduce function counts sums from document attributes - there are about one million documents in view.
Here's the raw json of map / reduce function:
"Summary": {
"map": "function(doc) { if (doc.objectType == \"Document\" && (doc.severity !== 4)) { emit(doc.alertTime, {\"feeds\": doc.feeds, \"severity\": doc.severity}); } }",
"reduce": "function(keys, values, rereduce) {\n var i, j, k, v, res, feeds;\n res = {};\n if (!rereduce) {\n res[\"*,*\"] = values.length;\n for (i = 0; i < values.length; i++) {\n k = \"*,\"+values[i].severity;\n res[k] = (res[k] || 0) + 1;\n feeds = values[i].feeds;\n for (j = 0; j < feeds.length; j++) {\n k = feeds[j]+\",*\";\n res[k] = (res[k] || 0) + 1;\n k = feeds[j]+\",\"+values[i].severity;\n res[k] = (res[k] || 0) + 1;\n }\n }\n } else {\n for (i = 0; i < values.length; i++) {\n v = values[i];\n for (k in v) {\n res[k] = (res[k] || 0) + v[k];\n }\n }\n }\n return res;\n}\n"
}
View entries are like this:
{"id":"ac:431590","key":"2010-11-01T00:30:02.000+0000","value":{"feeds":["BNK"],"severity":2}}
Again, this works just fine after reboot (even with reduce) but seems to slow over time (about 24-48 hours) until it starts to timeout.
-Sami
On Mar 11, 2013, at 11:20 PM, Robert Newson <rn...@apache.org> wrote:
> Can you access /_log on the server? might have information.
>
> Does the custom reduce function actually reduce (that is, return a
> much smaller value than the input 'values' array)?
>
> B.
>
> On 11 March 2013 03:27, Sami Sierla <sa...@poplatek.fi> wrote:
>> Hi,
>>
>> There is custom reduce function in this view but result is same even if reduce=false. Could this have something to do with running out of file descriptors (I unfortunately don't have sufficient user rights to check this on server). Issue began after we added two databases (unrelated to this view) to CouchDB (both have continuous replication to standby server).
>>
>> Best,
>> Sami
>>
>> On Mar 8, 2013, at 4:20 PM, Robert Newson <rn...@apache.org> wrote:
>>
>>> Do you have a custom reduce function? Are the keys/values in that view
>>> very large?
>>>
>>> On 8 March 2013 08:18, Sami Sierla <sa...@poplatek.fi> wrote:
>>>> Hi,
>>>>
>>>> We are having odd issue with CouchDB 1.2.0 view queries - queries to one particular view are very slow (tens of seconds with limit=1) but queries to other views in the same design doc perform as they should. Issue goes away if we restart CouchDB but comes back after few hours. Views are periodically refreshed so there are no indexing blocking the query. The setup we have consists of about ten CouchDB databases on active db server (RHEL 5) and second standby server with continuous CouchDB pull replication. Same view query works fine on the standby server that has no other processes running.
>>>>
>>>> Any ideas what could cause this?
>>>>
>>>> Best,
>>>> Sami Sierla
>>
Re: Very slow queries to view in design doc
Posted by Robert Newson <rn...@apache.org>.
Can you access /_log on the server? might have information.
Does the custom reduce function actually reduce (that is, return a
much smaller value than the input 'values' array)?
B.
On 11 March 2013 03:27, Sami Sierla <sa...@poplatek.fi> wrote:
> Hi,
>
> There is custom reduce function in this view but result is same even if reduce=false. Could this have something to do with running out of file descriptors (I unfortunately don't have sufficient user rights to check this on server). Issue began after we added two databases (unrelated to this view) to CouchDB (both have continuous replication to standby server).
>
> Best,
> Sami
>
> On Mar 8, 2013, at 4:20 PM, Robert Newson <rn...@apache.org> wrote:
>
>> Do you have a custom reduce function? Are the keys/values in that view
>> very large?
>>
>> On 8 March 2013 08:18, Sami Sierla <sa...@poplatek.fi> wrote:
>>> Hi,
>>>
>>> We are having odd issue with CouchDB 1.2.0 view queries - queries to one particular view are very slow (tens of seconds with limit=1) but queries to other views in the same design doc perform as they should. Issue goes away if we restart CouchDB but comes back after few hours. Views are periodically refreshed so there are no indexing blocking the query. The setup we have consists of about ten CouchDB databases on active db server (RHEL 5) and second standby server with continuous CouchDB pull replication. Same view query works fine on the standby server that has no other processes running.
>>>
>>> Any ideas what could cause this?
>>>
>>> Best,
>>> Sami Sierla
>
Re: Very slow queries to view in design doc
Posted by Sami Sierla <sa...@poplatek.fi>.
Hi,
There is custom reduce function in this view but result is same even if reduce=false. Could this have something to do with running out of file descriptors (I unfortunately don't have sufficient user rights to check this on server). Issue began after we added two databases (unrelated to this view) to CouchDB (both have continuous replication to standby server).
Best,
Sami
On Mar 8, 2013, at 4:20 PM, Robert Newson <rn...@apache.org> wrote:
> Do you have a custom reduce function? Are the keys/values in that view
> very large?
>
> On 8 March 2013 08:18, Sami Sierla <sa...@poplatek.fi> wrote:
>> Hi,
>>
>> We are having odd issue with CouchDB 1.2.0 view queries - queries to one particular view are very slow (tens of seconds with limit=1) but queries to other views in the same design doc perform as they should. Issue goes away if we restart CouchDB but comes back after few hours. Views are periodically refreshed so there are no indexing blocking the query. The setup we have consists of about ten CouchDB databases on active db server (RHEL 5) and second standby server with continuous CouchDB pull replication. Same view query works fine on the standby server that has no other processes running.
>>
>> Any ideas what could cause this?
>>
>> Best,
>> Sami Sierla
Re: Very slow queries to view in design doc
Posted by Robert Newson <rn...@apache.org>.
Do you have a custom reduce function? Are the keys/values in that view
very large?
On 8 March 2013 08:18, Sami Sierla <sa...@poplatek.fi> wrote:
> Hi,
>
> We are having odd issue with CouchDB 1.2.0 view queries - queries to one particular view are very slow (tens of seconds with limit=1) but queries to other views in the same design doc perform as they should. Issue goes away if we restart CouchDB but comes back after few hours. Views are periodically refreshed so there are no indexing blocking the query. The setup we have consists of about ten CouchDB databases on active db server (RHEL 5) and second standby server with continuous CouchDB pull replication. Same view query works fine on the standby server that has no other processes running.
>
> Any ideas what could cause this?
>
> Best,
> Sami Sierla