You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Luca Morandini <lm...@ieee.org> on 2015/02/10 03:38:26 UTC
Error while indexing a big JSON
Dear CouchDB community,
I am trying to process a large JSON (~11MB) with a view, but this operaiton fails:
[Fri, 07 Nov 2014 04:38:20 GMT] [info] [<0.3493.1>] Starting index update for db:
join88b760bd3b0346e09c7ca2d6c9a54215 idx: _design/join
[Fri, 07 Nov 2014 04:38:32 GMT] [error] [<0.3498.1>] OS Process Error <0.3206.1>
:: {os_process_error,
{exit_status,1}}
[Fri, 07 Nov 2014 04:38:34 GMT] [error] [emulator] Error in process <0.3498.1>
with exit value:
{{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"couch_os_process.erl"},{line,57}]},{couch_query_servers,map_doc_raw,2,[{file,"couch_query_servers.erl"},{line,88}]},{couch_mrview_updater,'-map_docs/2-fun-0-'...
I suppose Couchjs is finding itself short of memory: is this a sensible hypothesis ?
If so, how can I increase the memory devoted to the JavaScript view engine ?
Regards,
Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
[SOLVED] Re: Error while indexing a big JSON
Posted by Luca Morandini <lm...@ieee.org>.
On 10/02/15 13:38, Luca Morandini wrote:
> Dear CouchDB community,
>
> I am trying to process a large JSON (~11MB) with a view, but this operaiton fails:
This issue was solved by increasing the stack size, as described in
http://stackoverflow.com/questions/21273736/couchdb-views-os-process-error-big-documents
However, for some reasons, trying to parse the same JSON using couchjs directly
from the command line failed despite the increased stack size... I am a bit
puzzled, but all is well what ends well
Regards,
Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
Re: Error while indexing a big JSON
Posted by Luca Morandini <lm...@ieee.org>.
On 10/02/15 17:02, Wendall Cada wrote:
> You're running into the couchjs stack size limit, and the process is being killed.
Thanks, but I tried that already, to no avail:
couchjs -S 1073741824 large.js
SyntaxError: invalid label
Stacktrace:
@large.js:3
Failed to execute script.
(that's a 1GB stack).
Any other clues ?
Regards,
Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
Re: Error while indexing a big JSON
Posted by Wendall Cada <we...@apache.org>.
You're running into the couchjs stack size limit, and the process is
being killed. The message is a little obscure, but this can happen with
large JSON docs. There is a good answer left by Kxepal here:
http://stackoverflow.com/questions/21273736/couchdb-views-os-process-error-big-documents
on how to increase this.
HTH,
Wendall
On 02/09/2015 06:38 PM, Luca Morandini wrote:
> Dear CouchDB community,
>
> I am trying to process a large JSON (~11MB) with a view, but this
> operaiton fails:
>
> [Fri, 07 Nov 2014 04:38:20 GMT] [info] [<0.3493.1>] Starting index
> update for db: join88b760bd3b0346e09c7ca2d6c9a54215 idx: _design/join
> [Fri, 07 Nov 2014 04:38:32 GMT] [error] [<0.3498.1>] OS Process Error
> <0.3206.1> :: {os_process_error,
> {exit_status,1}}
> [Fri, 07 Nov 2014 04:38:34 GMT] [error] [emulator] Error in process
> <0.3498.1> with exit value:
> {{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"couch_os_process.erl"},{line,57}]},{couch_query_servers,map_doc_raw,2,[{file,"couch_query_servers.erl"},{line,88}]},{couch_mrview_updater,'-map_docs/2-fun-0-'...
>
>
> I suppose Couchjs is finding itself short of memory: is this a
> sensible hypothesis ?
>
> If so, how can I increase the memory devoted to the JavaScript view
> engine ?
>
> Regards,
>
> Luca Morandini
> Data Architect - AURIN project
> Melbourne eResearch Group
> Department of Computing and Information Systems
> University of Melbourne
> Tel. +61 03 903 58 380
> Skype: lmorandini
>
Re: Error while indexing a big JSON
Posted by Luca Morandini <lm...@ieee.org>.
On 10/02/15 13:44, Stanley Iriele wrote:
> One thing that van be happening is that you are hitting the RPC timeout for
> the view server.
I used to happen, but I fixed that.
> The real question is..why do you have an 11 mb file?.
> Would your project make sense it apart into smaller docs?
No, it is a geometry expressed into GeoJSON... no way I can sensibly split it.
Do you agree that this is a memory error (the error message is rather cryptic) ?
Thanks for your prompt reply,
Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
Re: Error while indexing a big JSON
Posted by Stanley Iriele <si...@gmail.com>.
One thing that van be happening is that you are hitting the RPC timeout for
the view server. The real question is..why do you have an 11 mb file?.
Would your project make sense it apart into smaller docs?
On Feb 9, 2015 6:41 PM, "Luca Morandini" <lm...@ieee.org> wrote:
> Dear CouchDB community,
>
> I am trying to process a large JSON (~11MB) with a view, but this
> operaiton fails:
>
> [Fri, 07 Nov 2014 04:38:20 GMT] [info] [<0.3493.1>] Starting index update
> for db: join88b760bd3b0346e09c7ca2d6c9a54215 idx: _design/join
> [Fri, 07 Nov 2014 04:38:32 GMT] [error] [<0.3498.1>] OS Process Error
> <0.3206.1> :: {os_process_error,
> {exit_status,1}}
> [Fri, 07 Nov 2014 04:38:34 GMT] [error] [emulator] Error in process
> <0.3498.1> with exit value: {{nocatch,{os_process_error,{
> exit_status,1}}},[{couch_os_process,prompt,2,[{file,"
> couch_os_process.erl"},{line,57}]},{couch_query_servers,
> map_doc_raw,2,[{file,"couch_query_servers.erl"},{line,88}]
> },{couch_mrview_updater,'-map_docs/2-fun-0-'...
>
> I suppose Couchjs is finding itself short of memory: is this a sensible
> hypothesis ?
>
> If so, how can I increase the memory devoted to the JavaScript view engine
> ?
>
> Regards,
>
> Luca Morandini
> Data Architect - AURIN project
> Melbourne eResearch Group
> Department of Computing and Information Systems
> University of Melbourne
> Tel. +61 03 903 58 380
> Skype: lmorandini
>
>