You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by david martin <da...@lymegreen.co.uk> on 2013/12/18 17:20:17 UTC
Timeout using Erlang views with large documents
Futon on Apache CouchDB 1.2 (according to Futon)
{"couchdb":"Welcome","version":"1.2.0"} according to ?
CouchDB 1.4.0 Ubuntu according to Package name
I set os_process_timeout 50000000000000 (effective infinity).
I ALWAYS get the VERY unhelpful message which merely prints the document contents.
Error: timeout % yes I know this but cannot do anything about it
{gen_server,call, % it's in a gen_server yes I know this!
[<0.14190.8>, % this is its PID yes I know this!
{prompt,[<<"map_doc">>, % it is a MAP function yes I know this!
{[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the document I am processing, Yes I know this!
{<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
.....
Yes it is a large and complex document (16K lines to make this happen on fast machine much less on Raspberry Pi).
Yes it uses Erlang view function.
Yes I DO want it to hog resources until it is finished.
Yes I am the administrator.
No I AM NOT INTERFERING WITH ANYTHING ELSE.
No I cannot dictate how big or small the document is.
Yes this is important to me.
I have not pursued this as I was using rcouch, I could not find the source of the timeout message.
I did not want to have to rebuild to fix this.
I did not want to bother the Couchdb team as I was using a fork of CouchDB.
Simlar issues have been raised and no answers forthcoming.
Mentions of "hidden tweaks", "this is not good for you", "have you got big documents" etc.
How do I get this NOT to timeout?
On rcouch I would change a value and rebuild a release to fix this (if I could identify the source).
If anybody can give a clue I will test their hypothesis and report back to the list.
--
David Martin
Re: Timeout using Erlang views with large documents
Posted by Robert Newson <rn...@apache.org>.
Yes, reconfirmed my finding. I added ?LOG_INFO lines to the
set_timeout clause in couch_native_server and it gets the current
os_process_timeout value. That's a bit silly (given it's not an os
process) but at least it's configurable. I stand by my original reply.
B.
On 18 December 2013 18:31, Robert Newson <rn...@apache.org> wrote:
> couch_native_server has the set_timeout callback, though. I'll re-test shortly.
>
> B.
>
>
> On 18 December 2013 18:17, Alexander Shorin <kx...@gmail.com> wrote:
>> iirc native query server has hardcoded timeout 5000 and ignores
>> os_process_timeout setting.
>> --
>> ,,,^..^,,,
>>
>>
>> On Wed, Dec 18, 2013 at 10:05 PM, Robert Newson <rn...@apache.org> wrote:
>>> I've confirmed that the native view server honors that timeout, can
>>> you tell me what;
>>>
>>> curl localhost:5984/_config/couchdb/os_process_timeout
>>>
>>> returns? You might need to bounce couchdb in any case, as it applies
>>> this timeout setting when it creates the process, and we keep a pool
>>> of them around, so changes to timeout after that won't be picked up
>>> until they're rebuild. restarting couchdb is the quickest way to
>>> ensure that.
>>>
>>> B.
>>>
>>>
>>> On 18 December 2013 16:20, david martin <da...@lymegreen.co.uk> wrote:
>>>> Futon on Apache CouchDB 1.2 (according to Futon)
>>>> {"couchdb":"Welcome","version":"1.2.0"} according to ?
>>>> CouchDB 1.4.0 Ubuntu according to Package name
>>>>
>>>> I set os_process_timeout 50000000000000 (effective infinity).
>>>>
>>>> I ALWAYS get the VERY unhelpful message which merely prints the document
>>>> contents.
>>>>
>>>> Error: timeout % yes I know this but cannot do anything about it
>>>>
>>>> {gen_server,call, % it's in a gen_server yes I know this!
>>>> [<0.14190.8>, % this is its PID yes I know this!
>>>> {prompt,[<<"map_doc">>, % it is a MAP function yes I know
>>>> this!
>>>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the document I
>>>> am processing, Yes I know this!
>>>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>>>> .....
>>>>
>>>> Yes it is a large and complex document (16K lines to make this happen on
>>>> fast machine much less on Raspberry Pi).
>>>> Yes it uses Erlang view function.
>>>> Yes I DO want it to hog resources until it is finished.
>>>> Yes I am the administrator.
>>>> No I AM NOT INTERFERING WITH ANYTHING ELSE.
>>>> No I cannot dictate how big or small the document is.
>>>> Yes this is important to me.
>>>> I have not pursued this as I was using rcouch, I could not find the source
>>>> of the timeout message.
>>>> I did not want to have to rebuild to fix this.
>>>> I did not want to bother the Couchdb team as I was using a fork of CouchDB.
>>>> Simlar issues have been raised and no answers forthcoming.
>>>> Mentions of "hidden tweaks", "this is not good for you", "have you got big
>>>> documents" etc.
>>>>
>>>> How do I get this NOT to timeout?
>>>>
>>>> On rcouch I would change a value and rebuild a release to fix this (if I
>>>> could identify the source).
>>>> If anybody can give a clue I will test their hypothesis and report back to
>>>> the list.
>>>>
>>>> --
>>>> David Martin
>>>>
Re: Timeout using Erlang views with large documents
Posted by Robert Newson <rn...@apache.org>.
couch_native_server has the set_timeout callback, though. I'll re-test shortly.
B.
On 18 December 2013 18:17, Alexander Shorin <kx...@gmail.com> wrote:
> iirc native query server has hardcoded timeout 5000 and ignores
> os_process_timeout setting.
> --
> ,,,^..^,,,
>
>
> On Wed, Dec 18, 2013 at 10:05 PM, Robert Newson <rn...@apache.org> wrote:
>> I've confirmed that the native view server honors that timeout, can
>> you tell me what;
>>
>> curl localhost:5984/_config/couchdb/os_process_timeout
>>
>> returns? You might need to bounce couchdb in any case, as it applies
>> this timeout setting when it creates the process, and we keep a pool
>> of them around, so changes to timeout after that won't be picked up
>> until they're rebuild. restarting couchdb is the quickest way to
>> ensure that.
>>
>> B.
>>
>>
>> On 18 December 2013 16:20, david martin <da...@lymegreen.co.uk> wrote:
>>> Futon on Apache CouchDB 1.2 (according to Futon)
>>> {"couchdb":"Welcome","version":"1.2.0"} according to ?
>>> CouchDB 1.4.0 Ubuntu according to Package name
>>>
>>> I set os_process_timeout 50000000000000 (effective infinity).
>>>
>>> I ALWAYS get the VERY unhelpful message which merely prints the document
>>> contents.
>>>
>>> Error: timeout % yes I know this but cannot do anything about it
>>>
>>> {gen_server,call, % it's in a gen_server yes I know this!
>>> [<0.14190.8>, % this is its PID yes I know this!
>>> {prompt,[<<"map_doc">>, % it is a MAP function yes I know
>>> this!
>>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the document I
>>> am processing, Yes I know this!
>>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>>> .....
>>>
>>> Yes it is a large and complex document (16K lines to make this happen on
>>> fast machine much less on Raspberry Pi).
>>> Yes it uses Erlang view function.
>>> Yes I DO want it to hog resources until it is finished.
>>> Yes I am the administrator.
>>> No I AM NOT INTERFERING WITH ANYTHING ELSE.
>>> No I cannot dictate how big or small the document is.
>>> Yes this is important to me.
>>> I have not pursued this as I was using rcouch, I could not find the source
>>> of the timeout message.
>>> I did not want to have to rebuild to fix this.
>>> I did not want to bother the Couchdb team as I was using a fork of CouchDB.
>>> Simlar issues have been raised and no answers forthcoming.
>>> Mentions of "hidden tweaks", "this is not good for you", "have you got big
>>> documents" etc.
>>>
>>> How do I get this NOT to timeout?
>>>
>>> On rcouch I would change a value and rebuild a release to fix this (if I
>>> could identify the source).
>>> If anybody can give a clue I will test their hypothesis and report back to
>>> the list.
>>>
>>> --
>>> David Martin
>>>
Re: Timeout using Erlang views with large documents
Posted by Alexander Shorin <kx...@gmail.com>.
iirc native query server has hardcoded timeout 5000 and ignores
os_process_timeout setting.
--
,,,^..^,,,
On Wed, Dec 18, 2013 at 10:05 PM, Robert Newson <rn...@apache.org> wrote:
> I've confirmed that the native view server honors that timeout, can
> you tell me what;
>
> curl localhost:5984/_config/couchdb/os_process_timeout
>
> returns? You might need to bounce couchdb in any case, as it applies
> this timeout setting when it creates the process, and we keep a pool
> of them around, so changes to timeout after that won't be picked up
> until they're rebuild. restarting couchdb is the quickest way to
> ensure that.
>
> B.
>
>
> On 18 December 2013 16:20, david martin <da...@lymegreen.co.uk> wrote:
>> Futon on Apache CouchDB 1.2 (according to Futon)
>> {"couchdb":"Welcome","version":"1.2.0"} according to ?
>> CouchDB 1.4.0 Ubuntu according to Package name
>>
>> I set os_process_timeout 50000000000000 (effective infinity).
>>
>> I ALWAYS get the VERY unhelpful message which merely prints the document
>> contents.
>>
>> Error: timeout % yes I know this but cannot do anything about it
>>
>> {gen_server,call, % it's in a gen_server yes I know this!
>> [<0.14190.8>, % this is its PID yes I know this!
>> {prompt,[<<"map_doc">>, % it is a MAP function yes I know
>> this!
>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the document I
>> am processing, Yes I know this!
>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>> .....
>>
>> Yes it is a large and complex document (16K lines to make this happen on
>> fast machine much less on Raspberry Pi).
>> Yes it uses Erlang view function.
>> Yes I DO want it to hog resources until it is finished.
>> Yes I am the administrator.
>> No I AM NOT INTERFERING WITH ANYTHING ELSE.
>> No I cannot dictate how big or small the document is.
>> Yes this is important to me.
>> I have not pursued this as I was using rcouch, I could not find the source
>> of the timeout message.
>> I did not want to have to rebuild to fix this.
>> I did not want to bother the Couchdb team as I was using a fork of CouchDB.
>> Simlar issues have been raised and no answers forthcoming.
>> Mentions of "hidden tweaks", "this is not good for you", "have you got big
>> documents" etc.
>>
>> How do I get this NOT to timeout?
>>
>> On rcouch I would change a value and rebuild a release to fix this (if I
>> could identify the source).
>> If anybody can give a clue I will test their hypothesis and report back to
>> the list.
>>
>> --
>> David Martin
>>
Re: Timeout using Erlang views with large documents
Posted by Stanley Iriele <si...@gmail.com>.
Random asside.. Have you restarted the database?... Yes you can curl and
get the value but the process ..may not have respected it ...
On Dec 19, 2013 7:37 AM, "david martin" <da...@lymegreen.co.uk>
wrote:
> On 18/12/13 22:42, Robert Newson wrote:
>
>> "There is something hard coded in there and I will find it eventually
>> and find why it was put there and by whom."
>>
>> This attitude might discourage people from helping you with your efforts.
>>
>> B.
>>
>
> If a simple statement of intent dissuades people from helping, then I do
> not want their help.
>
> Your own attitude may need rearranging in the light of your response to a
> perfectly well directed question.
>
> There is only one conclusion I can draw from your rather trite response
> and that is you are
> getting too clever at put-downs and not clever enough at helpful answers.
>
> Please refrain from ever answering any questions I may ask this list.
>
> --
> David Martin
>
>
Re: Timeout using Erlang views with large documents
Posted by Robert Newson <rn...@apache.org>.
I filed https://issues.apache.org/jira/browse/COUCHDB-2013 for this.
The patch will be a little more involved than just changing the prompt function as the run method does not respect the timeout for many of its clauses. While changing the gen_server call to infinity is an easy fix it removes any upper limit on execution time of a map or reduce function. Perhaps that’s fine, maybe we allow native processes to take forever (in which case we should remove all the existing timeout plumbing), but I can’t quite convince myself of that.
On 20 Dec 2013, at 15:10, Adam Kocoloski <ko...@apache.org> wrote:
> Hey folks, back to the original question, the native process gen_server respects the timeout internally but the public API in the module still makes a gen_server:call with the default 5 second timeout:
>
> https://github.com/apache/couchdb/blob/1.5.0/src/couchdb/couch_native_process.erl#L62-L63
>
> Contrast this with the OS process version where it sets the timeout on the client call to infinity (thus leaving it to the server to control the flow):
>
> https://github.com/apache/couchdb/blob/1.5.0/src/couchdb/couch_os_process.erl#L51-L58
>
> Teaching the native_process API to do the same would be a welcome change. Is there a JIRA for this one already?
>
> Adam
>
Re: Timeout using Erlang views with large documents
Posted by Adam Kocoloski <ko...@apache.org>.
Hey folks, back to the original question, the native process gen_server respects the timeout internally but the public API in the module still makes a gen_server:call with the default 5 second timeout:
https://github.com/apache/couchdb/blob/1.5.0/src/couchdb/couch_native_process.erl#L62-L63
Contrast this with the OS process version where it sets the timeout on the client call to infinity (thus leaving it to the server to control the flow):
https://github.com/apache/couchdb/blob/1.5.0/src/couchdb/couch_os_process.erl#L51-L58
Teaching the native_process API to do the same would be a welcome change. Is there a JIRA for this one already?
Adam
Re: Timeout using Erlang views with large documents
Posted by meredrica <st...@meredrica.org>.
On 12/19/2013 04:37 PM, david martin wrote:
> On 18/12/13 22:42, Robert Newson wrote:
>> "There is something hard coded in there and I will find it eventually
>> and find why it was put there and by whom."
>>
>> This attitude might discourage people from helping you with your efforts.
>>
>> B.
>
> If a simple statement of intent dissuades people from helping, then I do
> not want their help.
>
> Your own attitude may need rearranging in the light of your response to
> a perfectly well directed question.
>
> There is only one conclusion I can draw from your rather trite response
> and that is you are
> getting too clever at put-downs and not clever enough at helpful answers.
>
> Please refrain from ever answering any questions I may ask this list.
>
I have to say your tone is not the best. The statement above sounds like
a threat. Also you realy have no idea who Robert is and what he does
with CouchDB, right?
Re: Timeout using Erlang views with large documents
Posted by david martin <da...@lymegreen.co.uk>.
On 18/12/13 22:42, Robert Newson wrote:
> "There is something hard coded in there and I will find it eventually
> and find why it was put there and by whom."
>
> This attitude might discourage people from helping you with your efforts.
>
> B.
If a simple statement of intent dissuades people from helping, then I do not want their help.
Your own attitude may need rearranging in the light of your response to a perfectly well directed question.
There is only one conclusion I can draw from your rather trite response and that is you are
getting too clever at put-downs and not clever enough at helpful answers.
Please refrain from ever answering any questions I may ask this list.
--
David Martin
Re: Timeout using Erlang views with large documents
Posted by david martin <da...@lymegreen.co.uk>.
On 19/12/13 10:43, Stanley Iriele wrote:
> moral of the story is
> you shouldn't take that long to process stuff
I would take less time if I parallelised the processing in my OpenCl Erlang NIF's
but too much hassle getting thro' roadblocks of arbitrary set timeouts.
This was to bridge the gap between implicit schemas vs explicit
ones.
--
David Martin
Re: Timeout using Erlang views with large documents
Posted by Stanley Iriele <si...@gmail.com>.
Haha..no worries... But just because the browser does it that way doesn't
mean you have to...you can even use an update handler to isolate your
logic... Instead of simply inserting it...like injecting type and what not.
I've been using couch db for a while... There has to be a way for that data
to be itemized in a more usable way....its OK for you the database admin to
not no the entire structure but you should expected behavior from the data
you store...basically it's an argument about Implicit schemas vs explicit
ones..... I digress though. When I said joins I meant take a bunch of
docs...traverse them and merge them together into a giant json... And send
it back in chunks... That should work just fine....moral of the story is
you shouldn't take that long to process stuff.. Are you sure your mail
function isn't doing something crazy?
Can you paste that if its not private?
Regards
On Dec 19, 2013 2:32 AM, "david martin" <da...@lymegreen.co.uk>
wrote:
> On 19/12/13 10:08, Stanley Iriele wrote:
>
>> Ahh yes yes you're right...hmm...it is possible that they place a guard on
>> the function and call it using the default value or something like
>> that...also...what is your doc so huge?... At that point its easier to
>> manage it in small docs and join them all in some view/ list function
>>
>
> In the scenario I am hypothetically looking at, you (the database creator
> and administrator)
> are not in control of the structure of the document, it may be of any form
> as long as it is JSON.
>
> I first came upon this analysing the output of Collusion (an analytic
> program that tracks all browsing behaviour /in the browser/).
> This JSON record becomes very large very quickly.
>
> So if the doc is big, it is big. I do not wish to pre-process it, If the
> browser can handle documents of this size surely CouchDB
> can manipulate them in its native language without being throttled
> arbitrarily.
>
> I also feel that the word join is not really what SQL-less is about.
>
> These are only opinions, and I am often wrong so Your Mileage May Vary
> (ymmv).
>
> On Dec 19, 2013 2:01 AM, "david martin" <da...@lymegreen.co.uk>
>> wrote:
>>
>> On 19/12/13 03:28, Stanley Iriele wrote:
>>>
>>> Why did you place quotes around your timeout?.... Its just the value...
>>>> No
>>>> quotes....
>>>>
>>>> The 'value' of the timeout "50000000000000" is merely the form or
>>> representation that is reported in the browser by curl.
>>>
>>> In Futon configuration it is also represented in the same way.
>>>
>>> In the Erlang core of CouchDB this string may be used and/or stored in
>>> ETS
>>> in a number of ways.
>>>
>>> An atom using list_to_atom/1,
>>> a list, Erlang strings are lists,
>>> a binary by using list_to_binary/1,
>>> a 'value' using list_to_integer/1 or list_to_float/1,
>>> a term using binary_to_term/1.
>>>
>>> So what you see is not necessarily what you get inside the core.
>>>
>>> Yes you are right that timeout is an integer value in gen_servers and a
>>> typical one is 5000 standing for 5 seconds.
>>>
>>> Thank you for illustrating this seeming anomaly.
>>>
>>>
>>>
>>> A
>>>
>>> On Dec 18, 2013 2:42 PM, "Robert Newson" <rn...@apache.org> wrote:
>>>>
>>>> "There is something hard coded in there and I will find it eventually
>>>>
>>>>> and find why it was put there and by whom."
>>>>>
>>>>> This attitude might discourage people from helping you with your
>>>>> efforts.
>>>>>
>>>>> B.
>>>>>
>>>>>
>>>>> On 18 December 2013 22:33, david martin <da...@lymegreen.co.uk>
>>>>> wrote:
>>>>>
>>>>> On 18/12/13 18:05, Robert Newson wrote:
>>>>>>
>>>>>> I've confirmed that the native view server honors that timeout, can
>>>>>>> you tell me what;
>>>>>>>
>>>>>>> curl localhost:5984/_config/couchdb/os_process_timeout
>>>>>>>
>>>>>>> restart CouchDB on 1.2 (latest in Ubuntu) then
>>>>>>
>>>>>> curl david:************@localhost:5984/_config/couchdb/os_
>>>>>> process_timeout
>>>>>> "50000000000000"
>>>>>> rerun gives
>>>>>> Error: timeout
>>>>>>
>>>>>> {gen_server,call,
>>>>>> [<0.200.0>,
>>>>>> {prompt,[<<"map_doc">>,
>>>>>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>},
>>>>>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>>>>>> {<<"test">>,
>>>>>> {[{<<"hey">>,
>>>>>> {[{<<"_id">>,
>>>>>> <<"61c3f496b9e4c8dc29b95270d9000370">>},........}
>>>>>>
>>>>>> Test JSON here ~16K lines
>>>>>>
>>>>>> https://friendpaste.com/6LkCbdENAe1gOZlD9DWCod
>>>>>>>
>>>>>>> Code as in couchdb/erlang list in "Using the Erlang view server to
>>>>>>
>>>>>> Educate
>>>>>
>>>>> in CouchDB"
>>>>>>
>>>>>> I have looked for this for some time hoping next release would fix it.
>>>>>> There is something hard coded in there and I will find it eventually
>>>>>> and
>>>>>> find why it was put there and by whom.
>>>>>>
>>>>>>
>>>>>>
>>>>>> returns? You might need to bounce couchdb in any case, as it applies
>>>>>>
>>>>>>> this timeout setting when it creates the process, and we keep a pool
>>>>>>> of them around, so changes to timeout after that won't be picked up
>>>>>>> until they're rebuild. restarting couchdb is the quickest way to
>>>>>>> ensure that.
>>>>>>>
>>>>>>> B.
>>>>>>>
>>>>>>>
>>>>>>> On 18 December 2013 16:20, david martin <
>>>>>>> david.martin@lymegreen.co.uk>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Futon on Apache CouchDB 1.2 (according to Futon)
>>>>>>>> {"couchdb":"Welcome","version":"1.2.0"} according to ?
>>>>>>>> CouchDB 1.4.0 Ubuntu according to Package name
>>>>>>>>
>>>>>>>> I set os_process_timeout 50000000000000 (effective infinity).
>>>>>>>>
>>>>>>>> I ALWAYS get the VERY unhelpful message which merely prints the
>>>>>>>> document
>>>>>>>> contents.
>>>>>>>>
>>>>>>>> Error: timeout % yes I know this but cannot do anything about
>>>>>>>> it
>>>>>>>>
>>>>>>>> {gen_server,call, % it's in a gen_server yes I know this!
>>>>>>>> [<0.14190.8>, % this is its PID yes I know this!
>>>>>>>> {prompt,[<<"map_doc">>, % it is a MAP function
>>>>>>>> yes I
>>>>>>>>
>>>>>>>> know
>>>>>>>
>>>>>> this!
>>>>>>
>>>>>>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the
>>>>>>>> document I
>>>>>>>> am processing, Yes I know this!
>>>>>>>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>>>>>>>> .....
>>>>>>>>
>>>>>>>> Yes it is a large and complex document (16K lines to make this
>>>>>>>> happen
>>>>>>>>
>>>>>>>> on
>>>>>>>
>>>>>> fast machine much less on Raspberry Pi).
>>>>>>
>>>>>>> Yes it uses Erlang view function.
>>>>>>>> Yes I DO want it to hog resources until it is finished.
>>>>>>>> Yes I am the administrator.
>>>>>>>> No I AM NOT INTERFERING WITH ANYTHING ELSE.
>>>>>>>> No I cannot dictate how big or small the document is.
>>>>>>>> Yes this is important to me.
>>>>>>>> I have not pursued this as I was using rcouch, I could not find the
>>>>>>>> source
>>>>>>>> of the timeout message.
>>>>>>>> I did not want to have to rebuild to fix this.
>>>>>>>> I did not want to bother the Couchdb team as I was using a fork of
>>>>>>>> CouchDB.
>>>>>>>> Simlar issues have been raised and no answers forthcoming.
>>>>>>>> Mentions of "hidden tweaks", "this is not good for you", "have you
>>>>>>>> got
>>>>>>>> big
>>>>>>>> documents" etc.
>>>>>>>>
>>>>>>>> How do I get this NOT to timeout?
>>>>>>>>
>>>>>>>> On rcouch I would change a value and rebuild a release to fix this
>>>>>>>> (if
>>>>>>>>
>>>>>>>> I
>>>>>>>
>>>>>> could identify the source).
>>>>>>
>>>>>>> If anybody can give a clue I will test their hypothesis and report
>>>>>>>> back
>>>>>>>> to
>>>>>>>> the list.
>>>>>>>>
>>>>>>>> --
>>>>>>>> David Martin
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>> David Martin
>>>>>>
>>>>>>
>>>>>> --
>>> David Martin
>>>
>>>
>>>
>
> --
> David Martin
>
>
Re: Timeout using Erlang views with large documents
Posted by david martin <da...@lymegreen.co.uk>.
On 19/12/13 10:08, Stanley Iriele wrote:
> Ahh yes yes you're right...hmm...it is possible that they place a guard on
> the function and call it using the default value or something like
> that...also...what is your doc so huge?... At that point its easier to
> manage it in small docs and join them all in some view/ list function
In the scenario I am hypothetically looking at, you (the database creator and administrator)
are not in control of the structure of the document, it may be of any form as long as it is JSON.
I first came upon this analysing the output of Collusion (an analytic program that tracks all browsing behaviour /in the browser/).
This JSON record becomes very large very quickly.
So if the doc is big, it is big. I do not wish to pre-process it, If the browser can handle documents of this size surely CouchDB
can manipulate them in its native language without being throttled arbitrarily.
I also feel that the word join is not really what SQL-less is about.
These are only opinions, and I am often wrong so Your Mileage May Vary (ymmv).
> On Dec 19, 2013 2:01 AM, "david martin" <da...@lymegreen.co.uk>
> wrote:
>
>> On 19/12/13 03:28, Stanley Iriele wrote:
>>
>>> Why did you place quotes around your timeout?.... Its just the value...
>>> No
>>> quotes....
>>>
>> The 'value' of the timeout "50000000000000" is merely the form or
>> representation that is reported in the browser by curl.
>>
>> In Futon configuration it is also represented in the same way.
>>
>> In the Erlang core of CouchDB this string may be used and/or stored in ETS
>> in a number of ways.
>>
>> An atom using list_to_atom/1,
>> a list, Erlang strings are lists,
>> a binary by using list_to_binary/1,
>> a 'value' using list_to_integer/1 or list_to_float/1,
>> a term using binary_to_term/1.
>>
>> So what you see is not necessarily what you get inside the core.
>>
>> Yes you are right that timeout is an integer value in gen_servers and a
>> typical one is 5000 standing for 5 seconds.
>>
>> Thank you for illustrating this seeming anomaly.
>>
>>
>>
>> A
>>
>>> On Dec 18, 2013 2:42 PM, "Robert Newson" <rn...@apache.org> wrote:
>>>
>>> "There is something hard coded in there and I will find it eventually
>>>> and find why it was put there and by whom."
>>>>
>>>> This attitude might discourage people from helping you with your efforts.
>>>>
>>>> B.
>>>>
>>>>
>>>> On 18 December 2013 22:33, david martin <da...@lymegreen.co.uk>
>>>> wrote:
>>>>
>>>>> On 18/12/13 18:05, Robert Newson wrote:
>>>>>
>>>>>> I've confirmed that the native view server honors that timeout, can
>>>>>> you tell me what;
>>>>>>
>>>>>> curl localhost:5984/_config/couchdb/os_process_timeout
>>>>>>
>>>>> restart CouchDB on 1.2 (latest in Ubuntu) then
>>>>>
>>>>> curl david:************@localhost:5984/_config/couchdb/os_
>>>>> process_timeout
>>>>> "50000000000000"
>>>>> rerun gives
>>>>> Error: timeout
>>>>>
>>>>> {gen_server,call,
>>>>> [<0.200.0>,
>>>>> {prompt,[<<"map_doc">>,
>>>>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>},
>>>>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>>>>> {<<"test">>,
>>>>> {[{<<"hey">>,
>>>>> {[{<<"_id">>,
>>>>> <<"61c3f496b9e4c8dc29b95270d9000370">>},........}
>>>>>
>>>>> Test JSON here ~16K lines
>>>>>
>>>>>> https://friendpaste.com/6LkCbdENAe1gOZlD9DWCod
>>>>>>
>>>>> Code as in couchdb/erlang list in "Using the Erlang view server to
>>>>>
>>>> Educate
>>>>
>>>>> in CouchDB"
>>>>>
>>>>> I have looked for this for some time hoping next release would fix it.
>>>>> There is something hard coded in there and I will find it eventually and
>>>>> find why it was put there and by whom.
>>>>>
>>>>>
>>>>>
>>>>> returns? You might need to bounce couchdb in any case, as it applies
>>>>>> this timeout setting when it creates the process, and we keep a pool
>>>>>> of them around, so changes to timeout after that won't be picked up
>>>>>> until they're rebuild. restarting couchdb is the quickest way to
>>>>>> ensure that.
>>>>>>
>>>>>> B.
>>>>>>
>>>>>>
>>>>>> On 18 December 2013 16:20, david martin <da...@lymegreen.co.uk>
>>>>>> wrote:
>>>>>>
>>>>>>> Futon on Apache CouchDB 1.2 (according to Futon)
>>>>>>> {"couchdb":"Welcome","version":"1.2.0"} according to ?
>>>>>>> CouchDB 1.4.0 Ubuntu according to Package name
>>>>>>>
>>>>>>> I set os_process_timeout 50000000000000 (effective infinity).
>>>>>>>
>>>>>>> I ALWAYS get the VERY unhelpful message which merely prints the
>>>>>>> document
>>>>>>> contents.
>>>>>>>
>>>>>>> Error: timeout % yes I know this but cannot do anything about it
>>>>>>>
>>>>>>> {gen_server,call, % it's in a gen_server yes I know this!
>>>>>>> [<0.14190.8>, % this is its PID yes I know this!
>>>>>>> {prompt,[<<"map_doc">>, % it is a MAP function yes I
>>>>>>>
>>>>>> know
>>>>> this!
>>>>>>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the
>>>>>>> document I
>>>>>>> am processing, Yes I know this!
>>>>>>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>>>>>>> .....
>>>>>>>
>>>>>>> Yes it is a large and complex document (16K lines to make this happen
>>>>>>>
>>>>>> on
>>>>> fast machine much less on Raspberry Pi).
>>>>>>> Yes it uses Erlang view function.
>>>>>>> Yes I DO want it to hog resources until it is finished.
>>>>>>> Yes I am the administrator.
>>>>>>> No I AM NOT INTERFERING WITH ANYTHING ELSE.
>>>>>>> No I cannot dictate how big or small the document is.
>>>>>>> Yes this is important to me.
>>>>>>> I have not pursued this as I was using rcouch, I could not find the
>>>>>>> source
>>>>>>> of the timeout message.
>>>>>>> I did not want to have to rebuild to fix this.
>>>>>>> I did not want to bother the Couchdb team as I was using a fork of
>>>>>>> CouchDB.
>>>>>>> Simlar issues have been raised and no answers forthcoming.
>>>>>>> Mentions of "hidden tweaks", "this is not good for you", "have you got
>>>>>>> big
>>>>>>> documents" etc.
>>>>>>>
>>>>>>> How do I get this NOT to timeout?
>>>>>>>
>>>>>>> On rcouch I would change a value and rebuild a release to fix this (if
>>>>>>>
>>>>>> I
>>>>> could identify the source).
>>>>>>> If anybody can give a clue I will test their hypothesis and report
>>>>>>> back
>>>>>>> to
>>>>>>> the list.
>>>>>>>
>>>>>>> --
>>>>>>> David Martin
>>>>>>>
>>>>>>>
>>>>> --
>>>>> David Martin
>>>>>
>>>>>
>> --
>> David Martin
>>
>>
--
David Martin
Re: Timeout using Erlang views with large documents
Posted by Stanley Iriele <si...@gmail.com>.
Ahh yes yes you're right...hmm...it is possible that they place a guard on
the function and call it using the default value or something like
that...also...what is your doc so huge?... At that point its easier to
manage it in small docs and join them all in some view/ list function
On Dec 19, 2013 2:01 AM, "david martin" <da...@lymegreen.co.uk>
wrote:
> On 19/12/13 03:28, Stanley Iriele wrote:
>
>> Why did you place quotes around your timeout?.... Its just the value...
>> No
>> quotes....
>>
>
> The 'value' of the timeout "50000000000000" is merely the form or
> representation that is reported in the browser by curl.
>
> In Futon configuration it is also represented in the same way.
>
> In the Erlang core of CouchDB this string may be used and/or stored in ETS
> in a number of ways.
>
> An atom using list_to_atom/1,
> a list, Erlang strings are lists,
> a binary by using list_to_binary/1,
> a 'value' using list_to_integer/1 or list_to_float/1,
> a term using binary_to_term/1.
>
> So what you see is not necessarily what you get inside the core.
>
> Yes you are right that timeout is an integer value in gen_servers and a
> typical one is 5000 standing for 5 seconds.
>
> Thank you for illustrating this seeming anomaly.
>
>
>
> A
>
>> On Dec 18, 2013 2:42 PM, "Robert Newson" <rn...@apache.org> wrote:
>>
>> "There is something hard coded in there and I will find it eventually
>>> and find why it was put there and by whom."
>>>
>>> This attitude might discourage people from helping you with your efforts.
>>>
>>> B.
>>>
>>>
>>> On 18 December 2013 22:33, david martin <da...@lymegreen.co.uk>
>>> wrote:
>>>
>>>> On 18/12/13 18:05, Robert Newson wrote:
>>>>
>>>>> I've confirmed that the native view server honors that timeout, can
>>>>> you tell me what;
>>>>>
>>>>> curl localhost:5984/_config/couchdb/os_process_timeout
>>>>>
>>>>
>>>> restart CouchDB on 1.2 (latest in Ubuntu) then
>>>>
>>>> curl david:************@localhost:5984/_config/couchdb/os_
>>>> process_timeout
>>>> "50000000000000"
>>>> rerun gives
>>>> Error: timeout
>>>>
>>>> {gen_server,call,
>>>> [<0.200.0>,
>>>> {prompt,[<<"map_doc">>,
>>>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>},
>>>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>>>> {<<"test">>,
>>>> {[{<<"hey">>,
>>>> {[{<<"_id">>,
>>>> <<"61c3f496b9e4c8dc29b95270d9000370">>},........}
>>>>
>>>> Test JSON here ~16K lines
>>>>
>>>>> https://friendpaste.com/6LkCbdENAe1gOZlD9DWCod
>>>>>
>>>> Code as in couchdb/erlang list in "Using the Erlang view server to
>>>>
>>> Educate
>>>
>>>> in CouchDB"
>>>>
>>>> I have looked for this for some time hoping next release would fix it.
>>>> There is something hard coded in there and I will find it eventually and
>>>> find why it was put there and by whom.
>>>>
>>>>
>>>>
>>>> returns? You might need to bounce couchdb in any case, as it applies
>>>>> this timeout setting when it creates the process, and we keep a pool
>>>>> of them around, so changes to timeout after that won't be picked up
>>>>> until they're rebuild. restarting couchdb is the quickest way to
>>>>> ensure that.
>>>>>
>>>>> B.
>>>>>
>>>>>
>>>>> On 18 December 2013 16:20, david martin <da...@lymegreen.co.uk>
>>>>> wrote:
>>>>>
>>>>>> Futon on Apache CouchDB 1.2 (according to Futon)
>>>>>> {"couchdb":"Welcome","version":"1.2.0"} according to ?
>>>>>> CouchDB 1.4.0 Ubuntu according to Package name
>>>>>>
>>>>>> I set os_process_timeout 50000000000000 (effective infinity).
>>>>>>
>>>>>> I ALWAYS get the VERY unhelpful message which merely prints the
>>>>>> document
>>>>>> contents.
>>>>>>
>>>>>> Error: timeout % yes I know this but cannot do anything about it
>>>>>>
>>>>>> {gen_server,call, % it's in a gen_server yes I know this!
>>>>>> [<0.14190.8>, % this is its PID yes I know this!
>>>>>> {prompt,[<<"map_doc">>, % it is a MAP function yes I
>>>>>>
>>>>> know
>>>
>>>> this!
>>>>>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the
>>>>>> document I
>>>>>> am processing, Yes I know this!
>>>>>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>>>>>> .....
>>>>>>
>>>>>> Yes it is a large and complex document (16K lines to make this happen
>>>>>>
>>>>> on
>>>
>>>> fast machine much less on Raspberry Pi).
>>>>>> Yes it uses Erlang view function.
>>>>>> Yes I DO want it to hog resources until it is finished.
>>>>>> Yes I am the administrator.
>>>>>> No I AM NOT INTERFERING WITH ANYTHING ELSE.
>>>>>> No I cannot dictate how big or small the document is.
>>>>>> Yes this is important to me.
>>>>>> I have not pursued this as I was using rcouch, I could not find the
>>>>>> source
>>>>>> of the timeout message.
>>>>>> I did not want to have to rebuild to fix this.
>>>>>> I did not want to bother the Couchdb team as I was using a fork of
>>>>>> CouchDB.
>>>>>> Simlar issues have been raised and no answers forthcoming.
>>>>>> Mentions of "hidden tweaks", "this is not good for you", "have you got
>>>>>> big
>>>>>> documents" etc.
>>>>>>
>>>>>> How do I get this NOT to timeout?
>>>>>>
>>>>>> On rcouch I would change a value and rebuild a release to fix this (if
>>>>>>
>>>>> I
>>>
>>>> could identify the source).
>>>>>> If anybody can give a clue I will test their hypothesis and report
>>>>>> back
>>>>>> to
>>>>>> the list.
>>>>>>
>>>>>> --
>>>>>> David Martin
>>>>>>
>>>>>>
>>>> --
>>>> David Martin
>>>>
>>>>
>
> --
> David Martin
>
>
Re: Timeout using Erlang views with large documents
Posted by david martin <da...@lymegreen.co.uk>.
On 19/12/13 03:28, Stanley Iriele wrote:
> Why did you place quotes around your timeout?.... Its just the value... No
> quotes....
The 'value' of the timeout "50000000000000" is merely the form or representation that is reported in the browser by curl.
In Futon configuration it is also represented in the same way.
In the Erlang core of CouchDB this string may be used and/or stored in ETS in a number of ways.
An atom using list_to_atom/1,
a list, Erlang strings are lists,
a binary by using list_to_binary/1,
a 'value' using list_to_integer/1 or list_to_float/1,
a term using binary_to_term/1.
So what you see is not necessarily what you get inside the core.
Yes you are right that timeout is an integer value in gen_servers and a typical one is 5000 standing for 5 seconds.
Thank you for illustrating this seeming anomaly.
A
> On Dec 18, 2013 2:42 PM, "Robert Newson" <rn...@apache.org> wrote:
>
>> "There is something hard coded in there and I will find it eventually
>> and find why it was put there and by whom."
>>
>> This attitude might discourage people from helping you with your efforts.
>>
>> B.
>>
>>
>> On 18 December 2013 22:33, david martin <da...@lymegreen.co.uk>
>> wrote:
>>> On 18/12/13 18:05, Robert Newson wrote:
>>>> I've confirmed that the native view server honors that timeout, can
>>>> you tell me what;
>>>>
>>>> curl localhost:5984/_config/couchdb/os_process_timeout
>>>
>>> restart CouchDB on 1.2 (latest in Ubuntu) then
>>>
>>> curl david:************@localhost:5984/_config/couchdb/os_process_timeout
>>> "50000000000000"
>>> rerun gives
>>> Error: timeout
>>>
>>> {gen_server,call,
>>> [<0.200.0>,
>>> {prompt,[<<"map_doc">>,
>>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>},
>>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>>> {<<"test">>,
>>> {[{<<"hey">>,
>>> {[{<<"_id">>,
>>> <<"61c3f496b9e4c8dc29b95270d9000370">>},........}
>>>
>>> Test JSON here ~16K lines
>>>> https://friendpaste.com/6LkCbdENAe1gOZlD9DWCod
>>> Code as in couchdb/erlang list in "Using the Erlang view server to
>> Educate
>>> in CouchDB"
>>>
>>> I have looked for this for some time hoping next release would fix it.
>>> There is something hard coded in there and I will find it eventually and
>>> find why it was put there and by whom.
>>>
>>>
>>>
>>>> returns? You might need to bounce couchdb in any case, as it applies
>>>> this timeout setting when it creates the process, and we keep a pool
>>>> of them around, so changes to timeout after that won't be picked up
>>>> until they're rebuild. restarting couchdb is the quickest way to
>>>> ensure that.
>>>>
>>>> B.
>>>>
>>>>
>>>> On 18 December 2013 16:20, david martin <da...@lymegreen.co.uk>
>>>> wrote:
>>>>> Futon on Apache CouchDB 1.2 (according to Futon)
>>>>> {"couchdb":"Welcome","version":"1.2.0"} according to ?
>>>>> CouchDB 1.4.0 Ubuntu according to Package name
>>>>>
>>>>> I set os_process_timeout 50000000000000 (effective infinity).
>>>>>
>>>>> I ALWAYS get the VERY unhelpful message which merely prints the
>>>>> document
>>>>> contents.
>>>>>
>>>>> Error: timeout % yes I know this but cannot do anything about it
>>>>>
>>>>> {gen_server,call, % it's in a gen_server yes I know this!
>>>>> [<0.14190.8>, % this is its PID yes I know this!
>>>>> {prompt,[<<"map_doc">>, % it is a MAP function yes I
>> know
>>>>> this!
>>>>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the
>>>>> document I
>>>>> am processing, Yes I know this!
>>>>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>>>>> .....
>>>>>
>>>>> Yes it is a large and complex document (16K lines to make this happen
>> on
>>>>> fast machine much less on Raspberry Pi).
>>>>> Yes it uses Erlang view function.
>>>>> Yes I DO want it to hog resources until it is finished.
>>>>> Yes I am the administrator.
>>>>> No I AM NOT INTERFERING WITH ANYTHING ELSE.
>>>>> No I cannot dictate how big or small the document is.
>>>>> Yes this is important to me.
>>>>> I have not pursued this as I was using rcouch, I could not find the
>>>>> source
>>>>> of the timeout message.
>>>>> I did not want to have to rebuild to fix this.
>>>>> I did not want to bother the Couchdb team as I was using a fork of
>>>>> CouchDB.
>>>>> Simlar issues have been raised and no answers forthcoming.
>>>>> Mentions of "hidden tweaks", "this is not good for you", "have you got
>>>>> big
>>>>> documents" etc.
>>>>>
>>>>> How do I get this NOT to timeout?
>>>>>
>>>>> On rcouch I would change a value and rebuild a release to fix this (if
>> I
>>>>> could identify the source).
>>>>> If anybody can give a clue I will test their hypothesis and report back
>>>>> to
>>>>> the list.
>>>>>
>>>>> --
>>>>> David Martin
>>>>>
>>>
>>> --
>>> David Martin
>>>
--
David Martin
Re: Timeout using Erlang views with large documents
Posted by Stanley Iriele <si...@gmail.com>.
Why did you place quotes around your timeout?.... Its just the value... No
quotes....
On Dec 18, 2013 2:42 PM, "Robert Newson" <rn...@apache.org> wrote:
> "There is something hard coded in there and I will find it eventually
> and find why it was put there and by whom."
>
> This attitude might discourage people from helping you with your efforts.
>
> B.
>
>
> On 18 December 2013 22:33, david martin <da...@lymegreen.co.uk>
> wrote:
> > On 18/12/13 18:05, Robert Newson wrote:
> >>
> >> I've confirmed that the native view server honors that timeout, can
> >> you tell me what;
> >>
> >> curl localhost:5984/_config/couchdb/os_process_timeout
> >
> >
> > restart CouchDB on 1.2 (latest in Ubuntu) then
> >
> > curl david:************@localhost:5984/_config/couchdb/os_process_timeout
> > "50000000000000"
> > rerun gives
> > Error: timeout
> >
> > {gen_server,call,
> > [<0.200.0>,
> > {prompt,[<<"map_doc">>,
> > {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>},
> > {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
> > {<<"test">>,
> > {[{<<"hey">>,
> > {[{<<"_id">>,
> > <<"61c3f496b9e4c8dc29b95270d9000370">>},........}
> >
> > Test JSON here ~16K lines
> >>
> >> https://friendpaste.com/6LkCbdENAe1gOZlD9DWCod
> >
> > Code as in couchdb/erlang list in "Using the Erlang view server to
> Educate
> > in CouchDB"
> >
> > I have looked for this for some time hoping next release would fix it.
> > There is something hard coded in there and I will find it eventually and
> > find why it was put there and by whom.
> >
> >
> >
> >>
> >> returns? You might need to bounce couchdb in any case, as it applies
> >> this timeout setting when it creates the process, and we keep a pool
> >> of them around, so changes to timeout after that won't be picked up
> >> until they're rebuild. restarting couchdb is the quickest way to
> >> ensure that.
> >>
> >> B.
> >>
> >>
> >> On 18 December 2013 16:20, david martin <da...@lymegreen.co.uk>
> >> wrote:
> >>>
> >>> Futon on Apache CouchDB 1.2 (according to Futon)
> >>> {"couchdb":"Welcome","version":"1.2.0"} according to ?
> >>> CouchDB 1.4.0 Ubuntu according to Package name
> >>>
> >>> I set os_process_timeout 50000000000000 (effective infinity).
> >>>
> >>> I ALWAYS get the VERY unhelpful message which merely prints the
> >>> document
> >>> contents.
> >>>
> >>> Error: timeout % yes I know this but cannot do anything about it
> >>>
> >>> {gen_server,call, % it's in a gen_server yes I know this!
> >>> [<0.14190.8>, % this is its PID yes I know this!
> >>> {prompt,[<<"map_doc">>, % it is a MAP function yes I
> know
> >>> this!
> >>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the
> >>> document I
> >>> am processing, Yes I know this!
> >>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
> >>> .....
> >>>
> >>> Yes it is a large and complex document (16K lines to make this happen
> on
> >>> fast machine much less on Raspberry Pi).
> >>> Yes it uses Erlang view function.
> >>> Yes I DO want it to hog resources until it is finished.
> >>> Yes I am the administrator.
> >>> No I AM NOT INTERFERING WITH ANYTHING ELSE.
> >>> No I cannot dictate how big or small the document is.
> >>> Yes this is important to me.
> >>> I have not pursued this as I was using rcouch, I could not find the
> >>> source
> >>> of the timeout message.
> >>> I did not want to have to rebuild to fix this.
> >>> I did not want to bother the Couchdb team as I was using a fork of
> >>> CouchDB.
> >>> Simlar issues have been raised and no answers forthcoming.
> >>> Mentions of "hidden tweaks", "this is not good for you", "have you got
> >>> big
> >>> documents" etc.
> >>>
> >>> How do I get this NOT to timeout?
> >>>
> >>> On rcouch I would change a value and rebuild a release to fix this (if
> I
> >>> could identify the source).
> >>> If anybody can give a clue I will test their hypothesis and report back
> >>> to
> >>> the list.
> >>>
> >>> --
> >>> David Martin
> >>>
> >>
> >
> >
> > --
> > David Martin
> >
>
Re: Timeout using Erlang views with large documents
Posted by Robert Newson <rn...@apache.org>.
"There is something hard coded in there and I will find it eventually
and find why it was put there and by whom."
This attitude might discourage people from helping you with your efforts.
B.
On 18 December 2013 22:33, david martin <da...@lymegreen.co.uk> wrote:
> On 18/12/13 18:05, Robert Newson wrote:
>>
>> I've confirmed that the native view server honors that timeout, can
>> you tell me what;
>>
>> curl localhost:5984/_config/couchdb/os_process_timeout
>
>
> restart CouchDB on 1.2 (latest in Ubuntu) then
>
> curl david:************@localhost:5984/_config/couchdb/os_process_timeout
> "50000000000000"
> rerun gives
> Error: timeout
>
> {gen_server,call,
> [<0.200.0>,
> {prompt,[<<"map_doc">>,
> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>},
> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
> {<<"test">>,
> {[{<<"hey">>,
> {[{<<"_id">>,
> <<"61c3f496b9e4c8dc29b95270d9000370">>},........}
>
> Test JSON here ~16K lines
>>
>> https://friendpaste.com/6LkCbdENAe1gOZlD9DWCod
>
> Code as in couchdb/erlang list in "Using the Erlang view server to Educate
> in CouchDB"
>
> I have looked for this for some time hoping next release would fix it.
> There is something hard coded in there and I will find it eventually and
> find why it was put there and by whom.
>
>
>
>>
>> returns? You might need to bounce couchdb in any case, as it applies
>> this timeout setting when it creates the process, and we keep a pool
>> of them around, so changes to timeout after that won't be picked up
>> until they're rebuild. restarting couchdb is the quickest way to
>> ensure that.
>>
>> B.
>>
>>
>> On 18 December 2013 16:20, david martin <da...@lymegreen.co.uk>
>> wrote:
>>>
>>> Futon on Apache CouchDB 1.2 (according to Futon)
>>> {"couchdb":"Welcome","version":"1.2.0"} according to ?
>>> CouchDB 1.4.0 Ubuntu according to Package name
>>>
>>> I set os_process_timeout 50000000000000 (effective infinity).
>>>
>>> I ALWAYS get the VERY unhelpful message which merely prints the
>>> document
>>> contents.
>>>
>>> Error: timeout % yes I know this but cannot do anything about it
>>>
>>> {gen_server,call, % it's in a gen_server yes I know this!
>>> [<0.14190.8>, % this is its PID yes I know this!
>>> {prompt,[<<"map_doc">>, % it is a MAP function yes I know
>>> this!
>>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the
>>> document I
>>> am processing, Yes I know this!
>>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>>> .....
>>>
>>> Yes it is a large and complex document (16K lines to make this happen on
>>> fast machine much less on Raspberry Pi).
>>> Yes it uses Erlang view function.
>>> Yes I DO want it to hog resources until it is finished.
>>> Yes I am the administrator.
>>> No I AM NOT INTERFERING WITH ANYTHING ELSE.
>>> No I cannot dictate how big or small the document is.
>>> Yes this is important to me.
>>> I have not pursued this as I was using rcouch, I could not find the
>>> source
>>> of the timeout message.
>>> I did not want to have to rebuild to fix this.
>>> I did not want to bother the Couchdb team as I was using a fork of
>>> CouchDB.
>>> Simlar issues have been raised and no answers forthcoming.
>>> Mentions of "hidden tweaks", "this is not good for you", "have you got
>>> big
>>> documents" etc.
>>>
>>> How do I get this NOT to timeout?
>>>
>>> On rcouch I would change a value and rebuild a release to fix this (if I
>>> could identify the source).
>>> If anybody can give a clue I will test their hypothesis and report back
>>> to
>>> the list.
>>>
>>> --
>>> David Martin
>>>
>>
>
>
> --
> David Martin
>
Re: Timeout using Erlang views with large documents
Posted by david martin <da...@lymegreen.co.uk>.
On 18/12/13 18:05, Robert Newson wrote:
> I've confirmed that the native view server honors that timeout, can
> you tell me what;
>
> curl localhost:5984/_config/couchdb/os_process_timeout
restart CouchDB on 1.2 (latest in Ubuntu) then
curl david:************@localhost:5984/_config/couchdb/os_process_timeout
"50000000000000"
rerun gives
Error: timeout
{gen_server,call,
[<0.200.0>,
{prompt,[<<"map_doc">>,
{[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>},
{<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
{<<"test">>,
{[{<<"hey">>,
{[{<<"_id">>,
<<"61c3f496b9e4c8dc29b95270d9000370">>},........}
Test JSON here ~16K lines
> https://friendpaste.com/6LkCbdENAe1gOZlD9DWCod
Code as in couchdb/erlang list in "Using the Erlang view server to Educate in CouchDB"
I have looked for this for some time hoping next release would fix it.
There is something hard coded in there and I will find it eventually and find why it was put there and by whom.
>
> returns? You might need to bounce couchdb in any case, as it applies
> this timeout setting when it creates the process, and we keep a pool
> of them around, so changes to timeout after that won't be picked up
> until they're rebuild. restarting couchdb is the quickest way to
> ensure that.
>
> B.
>
>
> On 18 December 2013 16:20, david martin <da...@lymegreen.co.uk> wrote:
>> Futon on Apache CouchDB 1.2 (according to Futon)
>> {"couchdb":"Welcome","version":"1.2.0"} according to ?
>> CouchDB 1.4.0 Ubuntu according to Package name
>>
>> I set os_process_timeout 50000000000000 (effective infinity).
>>
>> I ALWAYS get the VERY unhelpful message which merely prints the document
>> contents.
>>
>> Error: timeout % yes I know this but cannot do anything about it
>>
>> {gen_server,call, % it's in a gen_server yes I know this!
>> [<0.14190.8>, % this is its PID yes I know this!
>> {prompt,[<<"map_doc">>, % it is a MAP function yes I know
>> this!
>> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the document I
>> am processing, Yes I know this!
>> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
>> .....
>>
>> Yes it is a large and complex document (16K lines to make this happen on
>> fast machine much less on Raspberry Pi).
>> Yes it uses Erlang view function.
>> Yes I DO want it to hog resources until it is finished.
>> Yes I am the administrator.
>> No I AM NOT INTERFERING WITH ANYTHING ELSE.
>> No I cannot dictate how big or small the document is.
>> Yes this is important to me.
>> I have not pursued this as I was using rcouch, I could not find the source
>> of the timeout message.
>> I did not want to have to rebuild to fix this.
>> I did not want to bother the Couchdb team as I was using a fork of CouchDB.
>> Simlar issues have been raised and no answers forthcoming.
>> Mentions of "hidden tweaks", "this is not good for you", "have you got big
>> documents" etc.
>>
>> How do I get this NOT to timeout?
>>
>> On rcouch I would change a value and rebuild a release to fix this (if I
>> could identify the source).
>> If anybody can give a clue I will test their hypothesis and report back to
>> the list.
>>
>> --
>> David Martin
>>
>
--
David Martin
Re: Timeout using Erlang views with large documents
Posted by Robert Newson <rn...@apache.org>.
I've confirmed that the native view server honors that timeout, can
you tell me what;
curl localhost:5984/_config/couchdb/os_process_timeout
returns? You might need to bounce couchdb in any case, as it applies
this timeout setting when it creates the process, and we keep a pool
of them around, so changes to timeout after that won't be picked up
until they're rebuild. restarting couchdb is the quickest way to
ensure that.
B.
On 18 December 2013 16:20, david martin <da...@lymegreen.co.uk> wrote:
> Futon on Apache CouchDB 1.2 (according to Futon)
> {"couchdb":"Welcome","version":"1.2.0"} according to ?
> CouchDB 1.4.0 Ubuntu according to Package name
>
> I set os_process_timeout 50000000000000 (effective infinity).
>
> I ALWAYS get the VERY unhelpful message which merely prints the document
> contents.
>
> Error: timeout % yes I know this but cannot do anything about it
>
> {gen_server,call, % it's in a gen_server yes I know this!
> [<0.14190.8>, % this is its PID yes I know this!
> {prompt,[<<"map_doc">>, % it is a MAP function yes I know
> this!
> {[{<<"_id">>,<<"61c3f496b9e4c8dc29b95270d9000370">>}, % it is the document I
> am processing, Yes I know this!
> {<<"_rev">>,<<"9-e48194151642345e0e3a4a5edfee56e4">>},
> .....
>
> Yes it is a large and complex document (16K lines to make this happen on
> fast machine much less on Raspberry Pi).
> Yes it uses Erlang view function.
> Yes I DO want it to hog resources until it is finished.
> Yes I am the administrator.
> No I AM NOT INTERFERING WITH ANYTHING ELSE.
> No I cannot dictate how big or small the document is.
> Yes this is important to me.
> I have not pursued this as I was using rcouch, I could not find the source
> of the timeout message.
> I did not want to have to rebuild to fix this.
> I did not want to bother the Couchdb team as I was using a fork of CouchDB.
> Simlar issues have been raised and no answers forthcoming.
> Mentions of "hidden tweaks", "this is not good for you", "have you got big
> documents" etc.
>
> How do I get this NOT to timeout?
>
> On rcouch I would change a value and rebuild a release to fix this (if I
> could identify the source).
> If anybody can give a clue I will test their hypothesis and report back to
> the list.
>
> --
> David Martin
>