You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Reddy B." <re...@live.fr> on 2016/11/23 15:06:32 UTC
Couchdb views crashing for large documents
Hello all !
Does somebody have any idea regarding the following ?
http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents
I've copied and pasted the question below for easy skimming :
up votedown votefavorite<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents#>
Couchdb keeps crashing whenever I try to build the index of the views of a design document emitting values for large documents. The total size of the database is 40 MB and I guess the documents are about 5 MB each. We're talking about large JSON without any attachment.
What concerns me is that I have 2.5 GB of free ram before trying to access the views but as soon as I try to access them, the CPU usage raises to 99% and all the free RAM gets eaten by erl.exebefore the indexing fails with exit code 1.
Here is the log:
[info] 2016-11-22T22:07:52.263000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.15603.334> died normal
[error] 2016-11-22T22:07:52.264000Z couchdb@localhost <0.15409.334> b9855eea74 rexi_server throw:{os_process_error,{exit_status,1}} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
Views skipping these documents can be accessed without issue. Which general guidelines could you provide me to help with this kind of situation? I am using couchdb 2.0 on windows.
Many thanks
Tried to increase the size of couchjs using various values for ./bin/couchjs -S MAX_RAM_TO_USE_IN_BYTES ./share/server/main.js without success.
I noticed that several instances of couchjs are started when I try to access the view perhaps, is there a way to restrict the number of concurrent couchjs instances to 1 ?
RE: Couchdb views crashing for large documents
Posted by "Reddy B." <re...@live.fr>.
Many thanks for your help, unfortunately I absolutely use these fields for indexing.
Basically we index chapters, paragraphs etc... within "Books" documents.
Remark : This results in large arrays within the book document, and - optimized - for loops within the views.
We need to stick to this design because business logic can get very complex, and actually even the views benefit from having access to a much required context.
What I can't explain to myself is why CouchDb would use 3Gb of ram to pass 1 document to couchjs (when I limit the number of instances of couchjs to 1). I sure expected this indexing scenario to be more resource intensive than usual, but it really feels like there is some sort of bug in CouchDb. It uses too much resources for a system reputed for operating well in low memory environment.
Our application uses a managed language, and the RAM memory overhead for instanciating the exact same books I'm trying to push to couchdb is about 150mb per book. How we go from that to 3Gb for a low-level database supposed to be optimized - and reputed to be so - is intriguing.
I'm trying to push 5, and I'm discovering that even pushing them one after the other turns out to not work as well as I hoped. Right now I'm getting {"error":"os_process_error","reason":"{exit_status,1}","ref":604603324} after 3s of indexing without clear meaning.
________________________________
De : Eli Stevens (Gmail) <wi...@gmail.com>
Envoyé : samedi 26 novembre 2016 04:36
À : CouchDB Developers
Objet : Re: Couchdb views crashing for large documents
When bitten by similar issues, I've found that moving large data
fields to attachments can work around the size issue. As long as you
don't need that data for indexing, it works well.
Cheers,
Eli
On Fri, Nov 25, 2016 at 2:10 AM, Reddy B. <re...@live.fr> wrote:
>
>
>
> I can see that erl.exe actually crashes during the indexing, can someone please tell me if the dump looks normal ?
>
>
> http://s000.tinyupload.com/?file_id=39457175902042589414
TinyUpload.com - best file hosting solution, with no limits, totaly free<http://s000.tinyupload.com/?file_id=39457175902042589414>
s000.tinyupload.com
TinyUpload.com - solution for tiny file hosting. No download limits, no upload limit. Totaly free.
>
>
>
> I don't know what to think about the :
>
> processes_used: 2477758424
>
> And more generally can't read between the lines yet with Erlang stuff. This is the first time CouchDb ever breaks for me, and as usual this kind of thing always happen at the worst time possible.
>
> These documents are basically books (storing them as one is better for design reasons). If I remove them all, and only post one book, the indexing goes without any issue. I can somehow live with that for time being - thanks God I own the product.... - but I cannot realistically expect that carefully indexing one big document at a time can be a mid-term solution for an application in production.
>
> I can recognize when I'm sitting on a bombshell, and I'm a bit concerned right now.
>
> ________________________________
> De : Reddy B. <re...@live.fr>
> Envoyé : mercredi 23 novembre 2016 16:06
> À : Reddy B.; dev@couchdb.apache.org
> Objet : Couchdb views crashing for large documents
>
>
> Hello all !
>
> Does somebody have any idea regarding the following ?
>
> http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents
[http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded&a]<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents>
Couchdb views crashing for large documents<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents>
stackoverflow.com
Couchdb keeps crashing whenever I try to build the index of the views of a design document emitting values for large documents. The total size of the database is 40 MB and I guess the documents are...
>
> [http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded&a]<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents>
>
> Couchdb views crashing for large documents<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents>
[http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded&a]<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents>
Couchdb views crashing for large documents<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents>
stackoverflow.com
Couchdb keeps crashing whenever I try to build the index of the views of a design document emitting values for large documents. The total size of the database is 40 MB and I guess the documents are...
> stackoverflow.com
> Couchdb keeps crashing whenever I try to build the index of the views of a design document emitting values for large documents. The total size of the database is 40 MB and I guess the documents are...
>
>
>
> I've copied and pasted the question below for easy skimming :
>
> up votedown votefavorite<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents#>
>
>
> Couchdb keeps crashing whenever I try to build the index of the views of a design document emitting values for large documents. The total size of the database is 40 MB and I guess the documents are about 5 MB each. We're talking about large JSON without any attachment.
>
> What concerns me is that I have 2.5 GB of free ram before trying to access the views but as soon as I try to access them, the CPU usage raises to 99% and all the free RAM gets eaten by erl.exebefore the indexing fails with exit code 1.
>
> Here is the log:
>
> [info] 2016-11-22T22:07:52.263000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.15603.334> died normal
>
> [error] 2016-11-22T22:07:52.264000Z couchdb@localhost <0.15409.334> b9855eea74 rexi_server throw:{os_process_error,{exit_status,1}} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
>
> Views skipping these documents can be accessed without issue. Which general guidelines could you provide me to help with this kind of situation? I am using couchdb 2.0 on windows.
>
> Many thanks
>
>
>
>
>
> Tried to increase the size of couchjs using various values for ./bin/couchjs -S MAX_RAM_TO_USE_IN_BYTES ./share/server/main.js without success.
>
>
> I noticed that several instances of couchjs are started when I try to access the view perhaps, is there a way to restrict the number of concurrent couchjs instances to 1 ?
>
>
>
>
>
Re: Couchdb views crashing for large documents
Posted by "Eli Stevens (Gmail)" <wi...@gmail.com>.
When bitten by similar issues, I've found that moving large data
fields to attachments can work around the size issue. As long as you
don't need that data for indexing, it works well.
Cheers,
Eli
On Fri, Nov 25, 2016 at 2:10 AM, Reddy B. <re...@live.fr> wrote:
>
>
>
> I can see that erl.exe actually crashes during the indexing, can someone please tell me if the dump looks normal ?
>
>
> http://s000.tinyupload.com/?file_id=39457175902042589414
>
>
>
> I don't know what to think about the :
>
> processes_used: 2477758424
>
> And more generally can't read between the lines yet with Erlang stuff. This is the first time CouchDb ever breaks for me, and as usual this kind of thing always happen at the worst time possible.
>
> These documents are basically books (storing them as one is better for design reasons). If I remove them all, and only post one book, the indexing goes without any issue. I can somehow live with that for time being - thanks God I own the product.... - but I cannot realistically expect that carefully indexing one big document at a time can be a mid-term solution for an application in production.
>
> I can recognize when I'm sitting on a bombshell, and I'm a bit concerned right now.
>
> ________________________________
> De : Reddy B. <re...@live.fr>
> Envoyé : mercredi 23 novembre 2016 16:06
> À : Reddy B.; dev@couchdb.apache.org
> Objet : Couchdb views crashing for large documents
>
>
> Hello all !
>
> Does somebody have any idea regarding the following ?
>
> http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents
>
> [http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded&a]<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents>
>
> Couchdb views crashing for large documents<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents>
> stackoverflow.com
> Couchdb keeps crashing whenever I try to build the index of the views of a design document emitting values for large documents. The total size of the database is 40 MB and I guess the documents are...
>
>
>
> I've copied and pasted the question below for easy skimming :
>
> up votedown votefavorite<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents#>
>
>
> Couchdb keeps crashing whenever I try to build the index of the views of a design document emitting values for large documents. The total size of the database is 40 MB and I guess the documents are about 5 MB each. We're talking about large JSON without any attachment.
>
> What concerns me is that I have 2.5 GB of free ram before trying to access the views but as soon as I try to access them, the CPU usage raises to 99% and all the free RAM gets eaten by erl.exebefore the indexing fails with exit code 1.
>
> Here is the log:
>
> [info] 2016-11-22T22:07:52.263000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.15603.334> died normal
>
> [error] 2016-11-22T22:07:52.264000Z couchdb@localhost <0.15409.334> b9855eea74 rexi_server throw:{os_process_error,{exit_status,1}} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
>
> Views skipping these documents can be accessed without issue. Which general guidelines could you provide me to help with this kind of situation? I am using couchdb 2.0 on windows.
>
> Many thanks
>
>
>
>
>
> Tried to increase the size of couchjs using various values for ./bin/couchjs -S MAX_RAM_TO_USE_IN_BYTES ./share/server/main.js without success.
>
>
> I noticed that several instances of couchjs are started when I try to access the view perhaps, is there a way to restrict the number of concurrent couchjs instances to 1 ?
>
>
>
>
>
RE: Couchdb views crashing for large documents
Posted by "Reddy B." <re...@live.fr>.
I can see that erl.exe actually crashes during the indexing, can someone please tell me if the dump looks normal ?
http://s000.tinyupload.com/?file_id=39457175902042589414
I don't know what to think about the :
processes_used: 2477758424
And more generally can't read between the lines yet with Erlang stuff. This is the first time CouchDb ever breaks for me, and as usual this kind of thing always happen at the worst time possible.
These documents are basically books (storing them as one is better for design reasons). If I remove them all, and only post one book, the indexing goes without any issue. I can somehow live with that for time being - thanks God I own the product.... - but I cannot realistically expect that carefully indexing one big document at a time can be a mid-term solution for an application in production.
I can recognize when I'm sitting on a bombshell, and I'm a bit concerned right now.
________________________________
De : Reddy B. <re...@live.fr>
Envoyé : mercredi 23 novembre 2016 16:06
À : Reddy B.; dev@couchdb.apache.org
Objet : Couchdb views crashing for large documents
Hello all !
Does somebody have any idea regarding the following ?
http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents
[http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded&a]<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents>
Couchdb views crashing for large documents<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents>
stackoverflow.com
Couchdb keeps crashing whenever I try to build the index of the views of a design document emitting values for large documents. The total size of the database is 40 MB and I guess the documents are...
I've copied and pasted the question below for easy skimming :
up votedown votefavorite<http://stackoverflow.com/questions/40752578/couchdb-views-crashing-for-large-documents#>
Couchdb keeps crashing whenever I try to build the index of the views of a design document emitting values for large documents. The total size of the database is 40 MB and I guess the documents are about 5 MB each. We're talking about large JSON without any attachment.
What concerns me is that I have 2.5 GB of free ram before trying to access the views but as soon as I try to access them, the CPU usage raises to 99% and all the free RAM gets eaten by erl.exebefore the indexing fails with exit code 1.
Here is the log:
[info] 2016-11-22T22:07:52.263000Z couchdb@localhost <0.212.0> -------- couch_proc_manager <0.15603.334> died normal
[error] 2016-11-22T22:07:52.264000Z couchdb@localhost <0.15409.334> b9855eea74 rexi_server throw:{os_process_error,{exit_status,1}} [{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,56}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,244}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
Views skipping these documents can be accessed without issue. Which general guidelines could you provide me to help with this kind of situation? I am using couchdb 2.0 on windows.
Many thanks
Tried to increase the size of couchjs using various values for ./bin/couchjs -S MAX_RAM_TO_USE_IN_BYTES ./share/server/main.js without success.
I noticed that several instances of couchjs are started when I try to access the view perhaps, is there a way to restrict the number of concurrent couchjs instances to 1 ?