You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Chris Stockton <ch...@gmail.com> on 2011/08/24 22:42:17 UTC
Our views keep failing with timeouts until a couchdb restart
We have about 3000 active databases on our server, about 200 of those
users (most active ones) keep randomly failing to load views. I am
having difficulty trying to decipher this stack trace to make any
sense of it, does anyone have any suggestions about what is happening
here? Are we just reaching some sort of capacity limit? The machines
are Intel(R) Xeon(R) CPU L5520 @ 2.27GHz x16 core with 50mb ram, we
are fine on disk space, and requests per second to be honest is around
1 with plenty of idle time between little spurts.
-Chris
----------------------------------------
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] 'GET'
/db_946/_design/views/_view/getByType?skip=0&key=["model","M4e551a78a9b3d"]&limit=25
{1,
1} from "WITHHELD"
Headers: [{'Accept',"*/*"},
{'Authorization',"Basic WITHHELD"},
{'Content-Type',"application/json; charset=utf-8"},
{'Host',"WITHHELD:5984"},
{'User-Agent',"PECL::HTTP/1.7.0 (PHP/5.3.4)"}]
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] OAuth Params:
[{"skip","0"},
{"key",
"[\"model\",\"M4e551a78a9b3d\"]"},
{"limit","25"}]
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] request_group
{Pid, Seq} {<0.25836.0>,1077}
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.82.0>] New task status for
db_946 _design/views: Processed 0 of 25 changes (0%)
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.82.0>] New task status for
db_946 _design/views: Finishing.
[Wed, 24 Aug 2011 15:36:35 GMT] [debug] [<0.31520.215>] 'GET'
/_all_dbs {1,1} from "WITHHELD"
Headers: [{'Accept',"*/*"},
{'Authorization',"Basic WITHHELD"},
{'Content-Type',"application/json; charset=utf-8"},
{'Host',"WITHHELD:5984"},
{'User-Agent',"PECL::HTTP/1.7.0 (PHP/5.3.4)"}]
[Wed, 24 Aug 2011 15:36:35 GMT] [debug] [<0.31520.215>] OAuth Params: []
[Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.25836.0>] Exit from linked
pid: {<0.27488.217>,
{timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,
<<"javascript">>}]}}}
[Wed, 24 Aug 2011 15:36:36 GMT] [error] [<0.25836.0>] ** Generic
server <0.25836.0> terminating
** Last message in was {'EXIT',<0.27488.217>,
{timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,<<"javascript">>}]}}}
** When Server state == {group_state,undefined,<<"db_946">>,
{"/WITHHELD/5984/views",<<"db_946">>,
{group,
<<88,143,236,54,199,129,241,187,65,118,56,84,121,73,
170,203>>,
nil,nil,<<"_design/views">>,<<"javascript">>,[],
[{view,0,0,0,
[<<"getByType">>],
<<"function(doc) {\n \n var sdoc =
eval(uneval(doc));\n if((typeof doc.model) !== \"undefined\") {\n
sdoc.data.rev = doc._rev;\n emit([doc.type, doc.model],
sdoc.data);\n } else if((typeof doc.type)
!== \"undefined\" && doc.type === \"couchimport\") {\n
emit([doc.type, doc.time], null);\n }\n}">>,
nil,[],[]}],
{[]},
nil,0,0,nil,nil}},
{group,
<<88,143,236,54,199,129,241,187,65,118,56,84,121,73,
170,203>>,
{db,<0.6703.0>,<0.6717.0>,nil,
<<"1313769124135037">>,<0.24135.71>,<0.24137.71>,
{db_header,5,1077,0,
{1011747,{410,120}},
{1014120,530},
{1022521,[]},
0,nil,nil,1000},
1077,
{btree,<0.24135.71>,
{1011747,{410,120}},
#Fun<couch_db_updater.10.118480628>,
#Fun<couch_db_updater.11.70256084>,
#Fun<couch_btree.5.49826377>,
#Fun<couch_db_updater.12.32819211>},
{btree,<0.24135.71>,
{1014120,530},
#Fun<couch_db_updater.13.25773661>,
#Fun<couch_db_updater.14.39338119>,
#Fun<couch_btree.5.49826377>,
#Fun<couch_db_updater.15.41342280>},
{btree,<0.24135.71>,
{1022521,[]},
#Fun<couch_btree.0.67022066>,
#Fun<couch_btree.1.72205677>,
#Fun<couch_btree.2.23775741>,nil},
1077,<<"db_946">>,
"/WITHHELD/5984/dbs/db_946.couch",[],[],
nil,
{user_ctx,null,[],undefined},
nil,1000,
[before_header,after_header,on_file_open],
false},
<0.24246.71>,<<"_design/views">>,<<"javascript">>,
[],
[{view,0,0,0,
[<<"getByType">>],
<<"function(doc) {\n \n var sdoc =
eval(uneval(doc));\n if((typeof doc.model) !== \"undefined\") {\n
sdoc.data.rev = doc._rev;\n emit([doc.type, doc.model],
sdoc.data);\n } else if((typeof doc.type)
!== \"undefined\" && doc.type === \"couchimport\") {\n
emit([doc.type, doc.time], null);\n }\n}">>,
{btree,<0.24246.71>,
{142166,{354,[]}},
#Fun<couch_btree.3.22452524>,
#Fun<couch_btree.4.103377371>,
#Fun<couch_view.less_json_ids.2>,
#Fun<couch_view_group.10.1717152>},
[],[]}],
{[]},
{btree,<0.24246.71>,
{30445,[]},
#Fun<couch_btree.0.67022066>,
#Fun<couch_btree.1.72205677>,
#Fun<couch_btree.2.23775741>,nil},
1051,0,nil,nil},
<0.27488.217>,nil,false,
[{{<0.14283.217>,#Ref<0.0.1341.210504>},1077}],
<0.24251.71>}
** Reason for termination ==
** {timeout,{gen_server,call,
[couch_query_servers,{get_proc,<<"javascript">>}]}}
[Wed, 24 Aug 2011 15:36:36 GMT] [error] [<0.25836.0>] {error_report,<0.30.0>,
{<0.25836.0>,crash_report,
[[{initial_call,
{couch_view_group,init,['Argument__1']}},
{pid,<0.25836.0>},
{registered_name,[]},
{error_info,
{exit,
{timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,<<"javascript">>}]}},
[{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,[<0.25835.0>]},
{messages,[]},
{links,[<0.24246.71>,<0.111.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,6765},
{stack_size,24},
{reductions,10266}],
[]]}}
[Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.14283.217>] request_group
Error {timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,<<"javascript">>}]}}
[Wed, 24 Aug 2011 15:36:36 GMT] [error] [<0.24246.71>] ** Generic
server <0.24246.71> terminating
** Last message in was {'EXIT',<0.25836.0>,
{timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,<<"javascript">>}]}}}
** When Server state == {file,
{file_descriptor,prim_file,{#Port<0.53260>,7389}},
0,143464}
** Reason for termination ==
** {timeout,{gen_server,call,
[couch_query_servers,{get_proc,<<"javascript">>}]}}
[Wed, 24 Aug 2011 15:36:36 GMT] [error] [<0.24246.71>] {error_report,<0.30.0>,
{<0.24246.71>,crash_report,
[[{initial_call,{couch_file,init,['Argument__1']}},
{pid,<0.24246.71>},
{registered_name,[]},
{error_info,
{exit,
{timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,<<"javascript">>}]}},
[{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,[<0.25836.0>,<0.25835.0>]},
{messages,[]},
{links,[<0.24251.71>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,2584},
{stack_size,24},
{reductions,17469}],
[{neighbour,
[{pid,<0.24251.71>},
{registered_name,[]},
{initial_call,
{couch_ref_counter,init,['Argument__1']}},
{current_function,{gen_server,loop,6}},
{ancestors,[<0.25836.0>,<0.25835.0>]},
{messages,[]},
{links,[<0.24246.71>]},
{dictionary,[]},
{trap_exit,false},
{status,waiting},
{heap_size,377},
{stack_size,9},
{reductions,141}]}]]}}
[Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.14283.217>] Minor error in
HTTP request: {timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,
<<"javascript">>}]}}
[Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.14283.217>] Stacktrace:
[{io_lib_pretty,cind_tag_tuple,7},
{io_lib_pretty,while_fail,3},
{io_lib_pretty,print,6},
{io_lib_format,build,3},
{io_lib_format,build,3},
{io_lib_format,build,3},
{io_lib_format,build,3},
{io_lib_format,build,3}]
[Wed, 24 Aug 2011 15:36:36 GMT] [info] [<0.14283.217>] WITHHELD - -
'GET' /db_946/_design/views/_view/getByType?skip=0&key=["model","M4e551a78a9b3d"]&limit=25
500
[Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.14283.217>] httpd 500
error response:
{"error":"timeout","reason":"{gen_server,call,[couch_query_servers,{get_proc,<<\"javascript\">>}]}"}
Re: Our views keep failing with timeouts until a couchdb restart
Posted by Randall Leeds <ra...@gmail.com>.
On Wed, Aug 24, 2011 at 20:30, Jason Smith <jh...@iriscouch.com> wrote:
> On Thu, Aug 25, 2011 at 9:55 AM, Paul Davis <pa...@gmail.com>
> wrote:
> >> 1. CouchDB has an odd, idiosyncratic, feature where GET queries
> >> produce side-effects. From HTTP, there are no side-effects, but as you
> >> can see, GETting a view can spawn couches processes and write files to
> >> the disk. Perhaps you could add ?stale=ok to all of your queries used
> >> in production. To my knowledge, stale=ok guarantees that couchjs will
> >> not be involved in servicing that query. This protects your users from
> >> seeing map/reduce errors. The down-side is that you must of course
> >> query the views yourself to keep them current.
>
As of CouchDB 1.1 you may use ?stale=update_after to trigger background
rebuilding but instantly stale data if the view is out of date.
Re: Our views keep failing with timeouts until a couchdb restart
Posted by Jason Smith <jh...@iriscouch.com>.
On Thu, Aug 25, 2011 at 9:55 AM, Paul Davis <pa...@gmail.com> wrote:
>> 1. CouchDB has an odd, idiosyncratic, feature where GET queries
>> produce side-effects. From HTTP, there are no side-effects, but as you
>> can see, GETting a view can spawn couches processes and write files to
>> the disk. Perhaps you could add ?stale=ok to all of your queries used
>> in production. To my knowledge, stale=ok guarantees that couchjs will
>> not be involved in servicing that query. This protects your users from
>> seeing map/reduce errors. The down-side is that you must of course
>> query the views yourself to keep them current.
>>
>
> No, couchjs will still be required to service reductions. stale=ok
> just means you don't have to wait for a possibly lengthy view build.
Thanks for the correction.
> Also, "GET's cause side effects" is a bit misleading I think. A GET on
> a view by default tries to return data from something equal to or
> newer than the current database update_seq. If the current view state
> hasn't caught up to the db's update_seq it waits. Classifying this as
> "GET's have side effects" is really like saying "GET to a Rails APP
> causes side effects because you have to wait for the template to
> render."
>
> For those familiar with internals its quite true that if the indexer
> isn't running, a new GET request will trigger an update. But say we
> change that so that view updates are tried every N seconds and a GET
> itself will never trigger an update but might endure a delay of N-1
> seconds before anything starts happening. You'd be hard pressed to say
> that the "GET caused side effects" in that case yet the observable
> behavior is the same: "sometimes it takes a while."
Yes, I use a sloppy definition. Every query has side effects, since
they are logged. But that is a trivial case. So you are right.
At any rate, for all the system administrator knows, a GET query can
create a new .view file and build a 500GB view which fills up the
disk. It's a small detail worth knowing. Day-to-day, it's not that
serious of a threat.
The main point is that you get more robust and more predictable
behavior by using ?stale=ok most of the time, and taking the trouble
to update the views independently. In my projects, I have a change
count and also time duration threshold. If either are exceeded then it
triggers a view refresh on all design documents for that database.
--
Iris Couch
Re: Our views keep failing with timeouts until a couchdb restart
Posted by Paul Davis <pa...@gmail.com>.
On Wed, Aug 24, 2011 at 9:41 PM, Jason Smith <jh...@iriscouch.com> wrote:
> (Migrated to user@)
>
> On Thu, Aug 25, 2011 at 4:05 AM, Chris Stockton
> <ch...@gmail.com> wrote:
>> Hello,
>>
>> On Wed, Aug 24, 2011 at 1:53 PM, Paul Davis <pa...@gmail.com> wrote:
>>> I bet you're hitting a bug we just recently fixed on trunk. Basically,
>>> there was a possibility that errors in some of the JS functions would
>>> end up causing a couchjs process to be come unusable without removing
>>> it from the list. Eventually there wouldn't be any spots left and
>>> clients would get timeouts like you see.
>>>
>>> Patch is at [1]. If it doesn't apply cleanly, you really only need the
>>> bits from couch_os_process.erl and couch_query_servers.erl. The rest
>>> is just test code.
>>>
>>> https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9
>>>
>>
>> Right after sending this I finally saw what the issue was, our bug
>> report here: https://issues.apache.org/jira/browse/COUCHDB-1257?focusedCommentId=13090484#comment-13090484
>> as a side effect was leaving lingering processes ultimately leading to
>> instability of couch.
>>
>> We are working to patch our reduce issue, and will look at applying
>> that commit perhaps once it hits mainline couch?
>
> Chris, I'm glad those bugs are identified and will be fixed soon. But
> in the meantime, perhaps you can change your code to add robustness? I
> can think of two ideas.
>
> 1. CouchDB has an odd, idiosyncratic, feature where GET queries
> produce side-effects. From HTTP, there are no side-effects, but as you
> can see, GETting a view can spawn couches processes and write files to
> the disk. Perhaps you could add ?stale=ok to all of your queries used
> in production. To my knowledge, stale=ok guarantees that couchjs will
> not be involved in servicing that query. This protects your users from
> seeing map/reduce errors. The down-side is that you must of course
> query the views yourself to keep them current.
>
No, couchjs will still be required to service reductions. stale=ok
just means you don't have to wait for a possibly lengthy view build.
Also, "GET's cause side effects" is a bit misleading I think. A GET on
a view by default tries to return data from something equal to or
newer than the current database update_seq. If the current view state
hasn't caught up to the db's update_seq it waits. Classifying this as
"GET's have side effects" is really like saying "GET to a Rails APP
causes side effects because you have to wait for the template to
render."
For those familiar with internals its quite true that if the indexer
isn't running, a new GET request will trigger an update. But say we
change that so that view updates are tried every N seconds and a GET
itself will never trigger an update but might endure a delay of N-1
seconds before anything starts happening. You'd be hard pressed to say
that the "GET caused side effects" in that case yet the observable
behavior is the same: "sometimes it takes a while."
> 2. Design documents should never be published (used in production)
> until their views are fully built. This is not a CouchDB bug, but
> rather a lack of tooling. The technique is pretty simple. Publish your
> design document under an alternative id: _design/example_staging.
> Next, query the views (which you are conveniently already doing in
> step 1!). When the views are fresh, with no bugs and everything looks
> good, query with HTTP COPY to promote _design/example_staging to
> _design/example. That is an atomic software upgrade; and views will be
> ready instantly. Not bad!
>
"Should never be ..." is a bit too prescriptive for my taste. Bottom
line, views can take a non trivial amount of time to build. Beware.
Though the advice for pre-building views and promoting is spot on if
you anticipate long view builds.
> Perhaps these ideas will help you work around your bugs. IMO, they are
> good general policies anyway.
>
> --
> Iris Couch
>
Re: Our views keep failing with timeouts until a couchdb restart
Posted by Jason Smith <jh...@iriscouch.com>.
(Migrated to user@)
On Thu, Aug 25, 2011 at 4:05 AM, Chris Stockton
<ch...@gmail.com> wrote:
> Hello,
>
> On Wed, Aug 24, 2011 at 1:53 PM, Paul Davis <pa...@gmail.com> wrote:
>> I bet you're hitting a bug we just recently fixed on trunk. Basically,
>> there was a possibility that errors in some of the JS functions would
>> end up causing a couchjs process to be come unusable without removing
>> it from the list. Eventually there wouldn't be any spots left and
>> clients would get timeouts like you see.
>>
>> Patch is at [1]. If it doesn't apply cleanly, you really only need the
>> bits from couch_os_process.erl and couch_query_servers.erl. The rest
>> is just test code.
>>
>> https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9
>>
>
> Right after sending this I finally saw what the issue was, our bug
> report here: https://issues.apache.org/jira/browse/COUCHDB-1257?focusedCommentId=13090484#comment-13090484
> as a side effect was leaving lingering processes ultimately leading to
> instability of couch.
>
> We are working to patch our reduce issue, and will look at applying
> that commit perhaps once it hits mainline couch?
Chris, I'm glad those bugs are identified and will be fixed soon. But
in the meantime, perhaps you can change your code to add robustness? I
can think of two ideas.
1. CouchDB has an odd, idiosyncratic, feature where GET queries
produce side-effects. From HTTP, there are no side-effects, but as you
can see, GETting a view can spawn couches processes and write files to
the disk. Perhaps you could add ?stale=ok to all of your queries used
in production. To my knowledge, stale=ok guarantees that couchjs will
not be involved in servicing that query. This protects your users from
seeing map/reduce errors. The down-side is that you must of course
query the views yourself to keep them current.
2. Design documents should never be published (used in production)
until their views are fully built. This is not a CouchDB bug, but
rather a lack of tooling. The technique is pretty simple. Publish your
design document under an alternative id: _design/example_staging.
Next, query the views (which you are conveniently already doing in
step 1!). When the views are fresh, with no bugs and everything looks
good, query with HTTP COPY to promote _design/example_staging to
_design/example. That is an atomic software upgrade; and views will be
ready instantly. Not bad!
Perhaps these ideas will help you work around your bugs. IMO, they are
good general policies anyway.
--
Iris Couch
Re: Our views keep failing with timeouts until a couchdb restart
Posted by Paul Davis <pa...@gmail.com>.
That link is to trunk. We should backport it to 1.1.x so it's in 1.1.1
as well. Perhaps even a 1.0.4.
On Wed, Aug 24, 2011 at 4:05 PM, Chris Stockton
<ch...@gmail.com> wrote:
> Hello,
>
> On Wed, Aug 24, 2011 at 1:53 PM, Paul Davis <pa...@gmail.com> wrote:
>> I bet you're hitting a bug we just recently fixed on trunk. Basically,
>> there was a possibility that errors in some of the JS functions would
>> end up causing a couchjs process to be come unusable without removing
>> it from the list. Eventually there wouldn't be any spots left and
>> clients would get timeouts like you see.
>>
>> Patch is at [1]. If it doesn't apply cleanly, you really only need the
>> bits from couch_os_process.erl and couch_query_servers.erl. The rest
>> is just test code.
>>
>> https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9
>>
>
> Right after sending this I finally saw what the issue was, our bug
> report here: https://issues.apache.org/jira/browse/COUCHDB-1257?focusedCommentId=13090484#comment-13090484
> as a side effect was leaving lingering processes ultimately leading to
> instability of couch.
>
> We are working to patch our reduce issue, and will look at applying
> that commit perhaps once it hits mainline couch?
>
> Thanks for the quick response,
>
> -Chris
>
Re: Our views keep failing with timeouts until a couchdb restart
Posted by Chris Stockton <ch...@gmail.com>.
Hello,
On Wed, Aug 24, 2011 at 1:53 PM, Paul Davis <pa...@gmail.com> wrote:
> I bet you're hitting a bug we just recently fixed on trunk. Basically,
> there was a possibility that errors in some of the JS functions would
> end up causing a couchjs process to be come unusable without removing
> it from the list. Eventually there wouldn't be any spots left and
> clients would get timeouts like you see.
>
> Patch is at [1]. If it doesn't apply cleanly, you really only need the
> bits from couch_os_process.erl and couch_query_servers.erl. The rest
> is just test code.
>
> https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9
>
Right after sending this I finally saw what the issue was, our bug
report here: https://issues.apache.org/jira/browse/COUCHDB-1257?focusedCommentId=13090484#comment-13090484
as a side effect was leaving lingering processes ultimately leading to
instability of couch.
We are working to patch our reduce issue, and will look at applying
that commit perhaps once it hits mainline couch?
Thanks for the quick response,
-Chris
Re: Our views keep failing with timeouts until a couchdb restart
Posted by Paul Davis <pa...@gmail.com>.
I bet you're hitting a bug we just recently fixed on trunk. Basically,
there was a possibility that errors in some of the JS functions would
end up causing a couchjs process to be come unusable without removing
it from the list. Eventually there wouldn't be any spots left and
clients would get timeouts like you see.
Patch is at [1]. If it doesn't apply cleanly, you really only need the
bits from couch_os_process.erl and couch_query_servers.erl. The rest
is just test code.
https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9
On Wed, Aug 24, 2011 at 3:42 PM, Chris Stockton
<ch...@gmail.com> wrote:
> We have about 3000 active databases on our server, about 200 of those
> users (most active ones) keep randomly failing to load views. I am
> having difficulty trying to decipher this stack trace to make any
> sense of it, does anyone have any suggestions about what is happening
> here? Are we just reaching some sort of capacity limit? The machines
> are Intel(R) Xeon(R) CPU L5520 @ 2.27GHz x16 core with 50mb ram, we
> are fine on disk space, and requests per second to be honest is around
> 1 with plenty of idle time between little spurts.
>
> -Chris
>
> ----------------------------------------
>
> [Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] 'GET'
> /db_946/_design/views/_view/getByType?skip=0&key=["model","M4e551a78a9b3d"]&limit=25
> {1,
>
> 1} from "WITHHELD"
> Headers: [{'Accept',"*/*"},
> {'Authorization',"Basic WITHHELD"},
> {'Content-Type',"application/json; charset=utf-8"},
> {'Host',"WITHHELD:5984"},
> {'User-Agent',"PECL::HTTP/1.7.0 (PHP/5.3.4)"}]
> [Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] OAuth Params:
> [{"skip","0"},
> {"key",
> "[\"model\",\"M4e551a78a9b3d\"]"},
> {"limit","25"}]
> [Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] request_group
> {Pid, Seq} {<0.25836.0>,1077}
> [Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.82.0>] New task status for
> db_946 _design/views: Processed 0 of 25 changes (0%)
> [Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.82.0>] New task status for
> db_946 _design/views: Finishing.
> [Wed, 24 Aug 2011 15:36:35 GMT] [debug] [<0.31520.215>] 'GET'
> /_all_dbs {1,1} from "WITHHELD"
> Headers: [{'Accept',"*/*"},
> {'Authorization',"Basic WITHHELD"},
> {'Content-Type',"application/json; charset=utf-8"},
> {'Host',"WITHHELD:5984"},
> {'User-Agent',"PECL::HTTP/1.7.0 (PHP/5.3.4)"}]
> [Wed, 24 Aug 2011 15:36:35 GMT] [debug] [<0.31520.215>] OAuth Params: []
> [Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.25836.0>] Exit from linked
> pid: {<0.27488.217>,
> {timeout,
> {gen_server,call,
> [couch_query_servers,
> {get_proc,
> <<"javascript">>}]}}}
> [Wed, 24 Aug 2011 15:36:36 GMT] [error] [<0.25836.0>] ** Generic
> server <0.25836.0> terminating
> ** Last message in was {'EXIT',<0.27488.217>,
> {timeout,
> {gen_server,call,
> [couch_query_servers,
> {get_proc,<<"javascript">>}]}}}
> ** When Server state == {group_state,undefined,<<"db_946">>,
> {"/WITHHELD/5984/views",<<"db_946">>,
> {group,
> <<88,143,236,54,199,129,241,187,65,118,56,84,121,73,
> 170,203>>,
> nil,nil,<<"_design/views">>,<<"javascript">>,[],
> [{view,0,0,0,
> [<<"getByType">>],
> <<"function(doc) {\n \n var sdoc =
> eval(uneval(doc));\n if((typeof doc.model) !== \"undefined\") {\n
> sdoc.data.rev = doc._rev;\n emit([doc.type, doc.model],
> sdoc.data);\n } else if((typeof doc.type)
> !== \"undefined\" && doc.type === \"couchimport\") {\n
> emit([doc.type, doc.time], null);\n }\n}">>,
> nil,[],[]}],
> {[]},
> nil,0,0,nil,nil}},
> {group,
> <<88,143,236,54,199,129,241,187,65,118,56,84,121,73,
> 170,203>>,
> {db,<0.6703.0>,<0.6717.0>,nil,
> <<"1313769124135037">>,<0.24135.71>,<0.24137.71>,
> {db_header,5,1077,0,
> {1011747,{410,120}},
> {1014120,530},
> {1022521,[]},
> 0,nil,nil,1000},
> 1077,
> {btree,<0.24135.71>,
> {1011747,{410,120}},
> #Fun<couch_db_updater.10.118480628>,
> #Fun<couch_db_updater.11.70256084>,
> #Fun<couch_btree.5.49826377>,
> #Fun<couch_db_updater.12.32819211>},
> {btree,<0.24135.71>,
> {1014120,530},
> #Fun<couch_db_updater.13.25773661>,
> #Fun<couch_db_updater.14.39338119>,
> #Fun<couch_btree.5.49826377>,
> #Fun<couch_db_updater.15.41342280>},
> {btree,<0.24135.71>,
> {1022521,[]},
> #Fun<couch_btree.0.67022066>,
> #Fun<couch_btree.1.72205677>,
> #Fun<couch_btree.2.23775741>,nil},
> 1077,<<"db_946">>,
> "/WITHHELD/5984/dbs/db_946.couch",[],[],
> nil,
> {user_ctx,null,[],undefined},
> nil,1000,
> [before_header,after_header,on_file_open],
> false},
> <0.24246.71>,<<"_design/views">>,<<"javascript">>,
> [],
> [{view,0,0,0,
> [<<"getByType">>],
> <<"function(doc) {\n \n var sdoc =
> eval(uneval(doc));\n if((typeof doc.model) !== \"undefined\") {\n
> sdoc.data.rev = doc._rev;\n emit([doc.type, doc.model],
> sdoc.data);\n } else if((typeof doc.type)
> !== \"undefined\" && doc.type === \"couchimport\") {\n
> emit([doc.type, doc.time], null);\n }\n}">>,
> {btree,<0.24246.71>,
> {142166,{354,[]}},
> #Fun<couch_btree.3.22452524>,
> #Fun<couch_btree.4.103377371>,
> #Fun<couch_view.less_json_ids.2>,
> #Fun<couch_view_group.10.1717152>},
> [],[]}],
> {[]},
> {btree,<0.24246.71>,
> {30445,[]},
> #Fun<couch_btree.0.67022066>,
> #Fun<couch_btree.1.72205677>,
> #Fun<couch_btree.2.23775741>,nil},
> 1051,0,nil,nil},
> <0.27488.217>,nil,false,
> [{{<0.14283.217>,#Ref<0.0.1341.210504>},1077}],
> <0.24251.71>}
> ** Reason for termination ==
> ** {timeout,{gen_server,call,
> [couch_query_servers,{get_proc,<<"javascript">>}]}}
>
> [Wed, 24 Aug 2011 15:36:36 GMT] [error] [<0.25836.0>] {error_report,<0.30.0>,
> {<0.25836.0>,crash_report,
> [[{initial_call,
> {couch_view_group,init,['Argument__1']}},
> {pid,<0.25836.0>},
> {registered_name,[]},
> {error_info,
> {exit,
> {timeout,
> {gen_server,call,
> [couch_query_servers,
> {get_proc,<<"javascript">>}]}},
> [{gen_server,terminate,6},
> {proc_lib,init_p_do_apply,3}]}},
> {ancestors,[<0.25835.0>]},
> {messages,[]},
> {links,[<0.24246.71>,<0.111.0>]},
> {dictionary,[]},
> {trap_exit,true},
> {status,running},
> {heap_size,6765},
> {stack_size,24},
> {reductions,10266}],
> []]}}
> [Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.14283.217>] request_group
> Error {timeout,
> {gen_server,call,
> [couch_query_servers,
> {get_proc,<<"javascript">>}]}}
> [Wed, 24 Aug 2011 15:36:36 GMT] [error] [<0.24246.71>] ** Generic
> server <0.24246.71> terminating
> ** Last message in was {'EXIT',<0.25836.0>,
> {timeout,
> {gen_server,call,
> [couch_query_servers,
> {get_proc,<<"javascript">>}]}}}
> ** When Server state == {file,
> {file_descriptor,prim_file,{#Port<0.53260>,7389}},
> 0,143464}
> ** Reason for termination ==
> ** {timeout,{gen_server,call,
> [couch_query_servers,{get_proc,<<"javascript">>}]}}
>
> [Wed, 24 Aug 2011 15:36:36 GMT] [error] [<0.24246.71>] {error_report,<0.30.0>,
> {<0.24246.71>,crash_report,
> [[{initial_call,{couch_file,init,['Argument__1']}},
> {pid,<0.24246.71>},
> {registered_name,[]},
> {error_info,
> {exit,
> {timeout,
> {gen_server,call,
> [couch_query_servers,
> {get_proc,<<"javascript">>}]}},
> [{gen_server,terminate,6},
> {proc_lib,init_p_do_apply,3}]}},
> {ancestors,[<0.25836.0>,<0.25835.0>]},
> {messages,[]},
> {links,[<0.24251.71>]},
> {dictionary,[]},
> {trap_exit,true},
> {status,running},
> {heap_size,2584},
> {stack_size,24},
> {reductions,17469}],
> [{neighbour,
> [{pid,<0.24251.71>},
> {registered_name,[]},
> {initial_call,
> {couch_ref_counter,init,['Argument__1']}},
> {current_function,{gen_server,loop,6}},
> {ancestors,[<0.25836.0>,<0.25835.0>]},
> {messages,[]},
> {links,[<0.24246.71>]},
> {dictionary,[]},
> {trap_exit,false},
> {status,waiting},
> {heap_size,377},
> {stack_size,9},
> {reductions,141}]}]]}}
> [Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.14283.217>] Minor error in
> HTTP request: {timeout,
> {gen_server,call,
> [couch_query_servers,
> {get_proc,
> <<"javascript">>}]}}
> [Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.14283.217>] Stacktrace:
> [{io_lib_pretty,cind_tag_tuple,7},
> {io_lib_pretty,while_fail,3},
> {io_lib_pretty,print,6},
> {io_lib_format,build,3},
> {io_lib_format,build,3},
> {io_lib_format,build,3},
> {io_lib_format,build,3},
> {io_lib_format,build,3}]
> [Wed, 24 Aug 2011 15:36:36 GMT] [info] [<0.14283.217>] WITHHELD - -
> 'GET' /db_946/_design/views/_view/getByType?skip=0&key=["model","M4e551a78a9b3d"]&limit=25
> 500
> [Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.14283.217>] httpd 500
> error response:
> {"error":"timeout","reason":"{gen_server,call,[couch_query_servers,{get_proc,<<\"javascript\">>}]}"}
>