You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Gordon Leland Hempton <gh...@gmail.com> on 2010/08/04 21:18:46 UTC
View Building Grinds System to Halt
I am working with a database of ~10 mil docs and 30GB size. I am trying to
build a view consisting of a single field as the key and a null document.
After creating the view and trying to access it, the couchdb beam.smp
command eventually goes up to 12.3GB virtual memory and uses 3.7GB of
physical memory. Memory is clearly the bottleneck as the CPU usage is around
5-15%. My system only has 4GB of memory so naturally all this paging is
slowing everything down to a crawl.
After a few minutes this usage will go back down, but it will get to the
same levels every time I access the view (which is still building and has
been for 12 hours).
Is this normal? How can I improve this?
--
Gordon L. Hempton
http://componentlab.com
http://hempton.com
Re: View Building Grinds System to Halt
Posted by Damien Katz <da...@apache.org>.
Can you post your design doc with the views in it? Keep in mind, CouchDB builds all views in a design doc at the same time, so it might not be the view you are accessing that's using up resources, but another included in the same design doc.
-Damien
On Aug 5, 2010, at 11:29 AM, Gordon Leland Hempton wrote:
> This also seems to happen whenever I cleanup views: beam.smp uses all the
> systems memory and then disappears, however the log message doesn't show up.
>
> Views work totally fine on databases that are small in size.
>
> Anyone have any recommendations?
>
> On Wed, Aug 4, 2010 at 4:07 PM, Gordon Leland Hempton <gh...@gmail.com>wrote:
>
>> After looking at the logs, it looks pretty bad:
>>
>> [Wed, 04 Aug 2010 22:57:28 GMT] [info] [<0.12883.0>] 127.0.0.1 - - 'GET'
>> /trademark/_design/Trademark/_view/by_filingDate?limit=10&stale=ok&descending=true&include_docs=true
>> 200
>>
>> [Wed, 04 Aug 2010 22:57:52 GMT] [error] [emulator] Error in process
>> <0.12957.0> with exit value: {badarg,[{erlang,binary_to_term,[<<1885695232
>> bytes>>]},{couch_file,pread_term,2},{couch_db,make_doc,5},{couch_db,open_doc_int,3},{couch_view_updater,load_doc,5},{couch_view_updater,'-update/2-fun-2-',8},{couch_btree,stream_kv_node2,8...
>>
>> Anyone know where to start?
>>
>> I am running Ubuntu 10.04 server with couchdb 1.0 installed from source. I
>> have made sure the xulrunner settings are correct.
>>
>> Thanks,
>>
>> On Wed, Aug 4, 2010 at 12:18 PM, Gordon Leland Hempton <ghempton@gmail.com
>>> wrote:
>>
>>> I am working with a database of ~10 mil docs and 30GB size. I am trying to
>>> build a view consisting of a single field as the key and a null document.
>>>
>>> After creating the view and trying to access it, the couchdb beam.smp
>>> command eventually goes up to 12.3GB virtual memory and uses 3.7GB of
>>> physical memory. Memory is clearly the bottleneck as the CPU usage is around
>>> 5-15%. My system only has 4GB of memory so naturally all this paging is
>>> slowing everything down to a crawl.
>>>
>>> After a few minutes this usage will go back down, but it will get to the
>>> same levels every time I access the view (which is still building and has
>>> been for 12 hours).
>>>
>>> Is this normal? How can I improve this?
>>>
>>> --
>>> Gordon L. Hempton
>>> http://componentlab.com
>>> http://hempton.com
>>>
>>
>>
>>
>> --
>> Gordon L. Hempton
>> http://componentlab.com
>> http://hempton.com
>>
>
>
>
> --
> Gordon L. Hempton
> http://componentlab.com
> http://hempton.com
Re: View Building Grinds System to Halt
Posted by Gordon Leland Hempton <gh...@gmail.com>.
This also seems to happen whenever I cleanup views: beam.smp uses all the
systems memory and then disappears, however the log message doesn't show up.
Views work totally fine on databases that are small in size.
Anyone have any recommendations?
On Wed, Aug 4, 2010 at 4:07 PM, Gordon Leland Hempton <gh...@gmail.com>wrote:
> After looking at the logs, it looks pretty bad:
>
> [Wed, 04 Aug 2010 22:57:28 GMT] [info] [<0.12883.0>] 127.0.0.1 - - 'GET'
> /trademark/_design/Trademark/_view/by_filingDate?limit=10&stale=ok&descending=true&include_docs=true
> 200
>
> [Wed, 04 Aug 2010 22:57:52 GMT] [error] [emulator] Error in process
> <0.12957.0> with exit value: {badarg,[{erlang,binary_to_term,[<<1885695232
> bytes>>]},{couch_file,pread_term,2},{couch_db,make_doc,5},{couch_db,open_doc_int,3},{couch_view_updater,load_doc,5},{couch_view_updater,'-update/2-fun-2-',8},{couch_btree,stream_kv_node2,8...
>
> Anyone know where to start?
>
> I am running Ubuntu 10.04 server with couchdb 1.0 installed from source. I
> have made sure the xulrunner settings are correct.
>
> Thanks,
>
> On Wed, Aug 4, 2010 at 12:18 PM, Gordon Leland Hempton <ghempton@gmail.com
> > wrote:
>
>> I am working with a database of ~10 mil docs and 30GB size. I am trying to
>> build a view consisting of a single field as the key and a null document.
>>
>> After creating the view and trying to access it, the couchdb beam.smp
>> command eventually goes up to 12.3GB virtual memory and uses 3.7GB of
>> physical memory. Memory is clearly the bottleneck as the CPU usage is around
>> 5-15%. My system only has 4GB of memory so naturally all this paging is
>> slowing everything down to a crawl.
>>
>> After a few minutes this usage will go back down, but it will get to the
>> same levels every time I access the view (which is still building and has
>> been for 12 hours).
>>
>> Is this normal? How can I improve this?
>>
>> --
>> Gordon L. Hempton
>> http://componentlab.com
>> http://hempton.com
>>
>
>
>
> --
> Gordon L. Hempton
> http://componentlab.com
> http://hempton.com
>
--
Gordon L. Hempton
http://componentlab.com
http://hempton.com
Re: View Building Grinds System to Halt
Posted by Damien Katz <da...@apache.org>.
What version of CouchDB are you running?
-Damien
On Aug 11, 2010, at 12:44 PM, Gordon Leland Hempton wrote:
> For what it's worth, the only way I was able to fix this was to delete the
> database and recreate all of the documents.
>
> On Fri, Aug 6, 2010 at 10:45 AM, Gordon Leland Hempton
> <gh...@gmail.com>wrote:
>
>> Thanks for your responses!
>>
>> The design document for this database is trivial. There is only the single
>> view which emits a single property as the key:
>>
>> {
>> "_id": "_design/Trademark",
>> "_rev": "6-ecab7bf9158e188ab2c39d37c4a1fb4d",
>> "views": {
>> "by_filingDate": {
>> "map": "function(doc) { emit(doc.filingDate, null); }"
>> }
>> }
>> }
>>
>>
>> This exact same setup works on a smaller scale of data on our development boxes.
>>
>>
>> I have also ensured that ldconfig was run. I am actually using the script provided on the wiki to update the xulrunner version. Futhermore, before I fixed xulrunner, NONE of my views were working, however now views are working fine on all databases exect ones of large size.
>>
>>
>> Thanks,
>>
>> Gordon
>>
>>
>> On Thu, Aug 5, 2010 at 1:25 PM, David Caylor <dc...@fh.org> wrote:
>>
>> Did you run ldconfig after you set the xulrunner settings? I did that to
>>> myself last time I updated a Ubuntu box where I'd installed couchdb from
>>> source. I updated the xulrunner.conf, but forgot to run ldconfig, and got
>>> pretty much what you are describing.
>>>
>>> David Caylor
>>>
>>>
>>> On Wed, Aug 4, 2010 at 4:07 PM, Gordon Leland Hempton <ghempton@gmail.com
>>>> wrote:
>>>
>>>> After looking at the logs, it looks pretty bad:
>>>>
>>>> [Wed, 04 Aug 2010 22:57:28 GMT] [info] [<0.12883.0>] 127.0.0.1 - - 'GET'
>>>>
>>>>
>>> /trademark/_design/Trademark/_view/by_filingDate?limit=10&stale=ok&descending=true&include_docs=true
>>>> 200
>>>>
>>>> [Wed, 04 Aug 2010 22:57:52 GMT] [error] [emulator] Error in process
>>>> <0.12957.0> with exit value:
>>> {badarg,[{erlang,binary_to_term,[<<1885695232
>>>>
>>>>
>>> bytes>>]},{couch_file,pread_term,2},{couch_db,make_doc,5},{couch_db,open_doc_int,3},{couch_view_updater,load_doc,5},{couch_view_updater,'-update/2-fun-2-',8},{couch_btree,stream_kv_node2,8...
>>>>
>>>> Anyone know where to start?
>>>>
>>>> I am running Ubuntu 10.04 server with couchdb 1.0 installed from source.
>>> I
>>>> have made sure the xulrunner settings are correct.
>>>>
>>>> Thanks,
>>>>
>>>> On Wed, Aug 4, 2010 at 12:18 PM, Gordon Leland Hempton
>>>> <gh...@gmail.com>wrote:
>>>>
>>>>> I am working with a database of ~10 mil docs and 30GB size. I am
>>> trying
>>>> to
>>>>> build a view consisting of a single field as the key and a null
>>> document.
>>>>>
>>>>> After creating the view and trying to access it, the couchdb beam.smp
>>>>> command eventually goes up to 12.3GB virtual memory and uses 3.7GB of
>>>>> physical memory. Memory is clearly the bottleneck as the CPU usage is
>>>> around
>>>>> 5-15%. My system only has 4GB of memory so naturally all this paging
>>> is
>>>>> slowing everything down to a crawl.
>>>>>
>>>>> After a few minutes this usage will go back down, but it will get to
>>> the
>>>>> same levels every time I access the view (which is still building and
>>> has
>>>>> been for 12 hours).
>>>>>
>>>>> Is this normal? How can I improve this?
>>>>>
>>>>> --
>>>>> Gordon L. Hempton
>>>>> http://componentlab.com
>>>>> http://hempton.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Gordon L. Hempton
>>>> http://componentlab.com
>>>> http://hempton.com
>>>>
>>>
>>
>>
>>
>> --
>> Gordon L. Hempton
>> http://componentlab.com
>> http://hempton.com
>>
>
>
>
> --
> Gordon L. Hempton
> http://componentlab.com
> http://hempton.com
Re: View Building Grinds System to Halt
Posted by Eric Carlson <er...@ericcarlson.co.uk>.
Not sure if this will help, but you did say at one point that you built
from source. Have a look in /usr/local/lib/couchdb/erlang/lib to see if
there is more than one couch-xxx folder. Each build adds its own
couch-xxx directory there, where xxx reflects the version number. If
there is more than one such directory, Erlang can become confused and do
some very strange things unless you remove all but the directory that
corresponds to the version of couchdb that you are running . The
simplest solution is to delete everything in
/usr/local/lib/couchdb/erlang/lib and to redo the 'sudo make install',
which will put back only the latest versions of everything. (This
assumes that you haven't added any Erlang files of your own to the
directory, of course.) Like I say, don't know if this is the problem,
but it's worth a look.
Although it is unrelated to your view problems, I would be remiss if I
didn't mention that if you are indeed using v1.0.0, you should read
http://couchdb.apache.org/notice/1.0.1.html.
Eric
On 11/08/10 20:44, Gordon Leland Hempton wrote:
> For what it's worth, the only way I was able to fix this was to delete the
> database and recreate all of the documents.
>
> On Fri, Aug 6, 2010 at 10:45 AM, Gordon Leland Hempton
> <gh...@gmail.com>wrote:
>
>> Thanks for your responses!
>>
>> The design document for this database is trivial. There is only the single
>> view which emits a single property as the key:
>>
>> {
>> "_id": "_design/Trademark",
>> "_rev": "6-ecab7bf9158e188ab2c39d37c4a1fb4d",
>> "views": {
>> "by_filingDate": {
>> "map": "function(doc) { emit(doc.filingDate, null); }"
>> }
>> }
>> }
>>
>>
>> This exact same setup works on a smaller scale of data on our development boxes.
>>
>>
>> I have also ensured that ldconfig was run. I am actually using the script provided on the wiki to update the xulrunner version. Futhermore, before I fixed xulrunner, NONE of my views were working, however now views are working fine on all databases exect ones of large size.
>>
>>
>> Thanks,
>>
>> Gordon
>>
>>
>> On Thu, Aug 5, 2010 at 1:25 PM, David Caylor <dc...@fh.org> wrote:
>>
>> Did you run ldconfig after you set the xulrunner settings? I did that to
>>> myself last time I updated a Ubuntu box where I'd installed couchdb from
>>> source. I updated the xulrunner.conf, but forgot to run ldconfig, and got
>>> pretty much what you are describing.
>>>
>>> David Caylor
>>>
>>>
>>> On Wed, Aug 4, 2010 at 4:07 PM, Gordon Leland Hempton <ghempton@gmail.com
>>>> wrote:
>>>> After looking at the logs, it looks pretty bad:
>>>>
>>>> [Wed, 04 Aug 2010 22:57:28 GMT] [info] [<0.12883.0>] 127.0.0.1 - - 'GET'
>>>>
>>>>
>>> /trademark/_design/Trademark/_view/by_filingDate?limit=10&stale=ok&descending=true&include_docs=true
>>>> 200
>>>>
>>>> [Wed, 04 Aug 2010 22:57:52 GMT] [error] [emulator] Error in process
>>>> <0.12957.0> with exit value:
>>> {badarg,[{erlang,binary_to_term,[<<1885695232
>>>>
>>> bytes>>]},{couch_file,pread_term,2},{couch_db,make_doc,5},{couch_db,open_doc_int,3},{couch_view_updater,load_doc,5},{couch_view_updater,'-update/2-fun-2-',8},{couch_btree,stream_kv_node2,8...
>>>> Anyone know where to start?
>>>>
>>>> I am running Ubuntu 10.04 server with couchdb 1.0 installed from source.
>>> I
>>>> have made sure the xulrunner settings are correct.
>>>>
>>>> Thanks,
>>>>
>>>> On Wed, Aug 4, 2010 at 12:18 PM, Gordon Leland Hempton
>>>> <gh...@gmail.com>wrote:
>>>>
>>>>> I am working with a database of ~10 mil docs and 30GB size. I am
>>> trying
>>>> to
>>>>> build a view consisting of a single field as the key and a null
>>> document.
>>>>> After creating the view and trying to access it, the couchdb beam.smp
>>>>> command eventually goes up to 12.3GB virtual memory and uses 3.7GB of
>>>>> physical memory. Memory is clearly the bottleneck as the CPU usage is
>>>> around
>>>>> 5-15%. My system only has 4GB of memory so naturally all this paging
>>> is
>>>>> slowing everything down to a crawl.
>>>>>
>>>>> After a few minutes this usage will go back down, but it will get to
>>> the
>>>>> same levels every time I access the view (which is still building and
>>> has
>>>>> been for 12 hours).
>>>>>
>>>>> Is this normal? How can I improve this?
>>>>>
>>>>> --
>>>>> Gordon L. Hempton
>>>>> http://componentlab.com
>>>>> http://hempton.com
>>>>>
>>>>
>>>>
>>>> --
>>>> Gordon L. Hempton
>>>> http://componentlab.com
>>>> http://hempton.com
>>>>
>>
>>
>> --
>> Gordon L. Hempton
>> http://componentlab.com
>> http://hempton.com
>>
>
>
Re: View Building Grinds System to Halt
Posted by Gordon Leland Hempton <gh...@gmail.com>.
For what it's worth, the only way I was able to fix this was to delete the
database and recreate all of the documents.
On Fri, Aug 6, 2010 at 10:45 AM, Gordon Leland Hempton
<gh...@gmail.com>wrote:
> Thanks for your responses!
>
> The design document for this database is trivial. There is only the single
> view which emits a single property as the key:
>
> {
> "_id": "_design/Trademark",
> "_rev": "6-ecab7bf9158e188ab2c39d37c4a1fb4d",
> "views": {
> "by_filingDate": {
> "map": "function(doc) { emit(doc.filingDate, null); }"
> }
> }
> }
>
>
> This exact same setup works on a smaller scale of data on our development boxes.
>
>
> I have also ensured that ldconfig was run. I am actually using the script provided on the wiki to update the xulrunner version. Futhermore, before I fixed xulrunner, NONE of my views were working, however now views are working fine on all databases exect ones of large size.
>
>
> Thanks,
>
> Gordon
>
>
> On Thu, Aug 5, 2010 at 1:25 PM, David Caylor <dc...@fh.org> wrote:
>
> Did you run ldconfig after you set the xulrunner settings? I did that to
>> myself last time I updated a Ubuntu box where I'd installed couchdb from
>> source. I updated the xulrunner.conf, but forgot to run ldconfig, and got
>> pretty much what you are describing.
>>
>> David Caylor
>>
>>
>> On Wed, Aug 4, 2010 at 4:07 PM, Gordon Leland Hempton <ghempton@gmail.com
>> >wrote:
>>
>> > After looking at the logs, it looks pretty bad:
>> >
>> > [Wed, 04 Aug 2010 22:57:28 GMT] [info] [<0.12883.0>] 127.0.0.1 - - 'GET'
>> >
>> >
>> /trademark/_design/Trademark/_view/by_filingDate?limit=10&stale=ok&descending=true&include_docs=true
>> > 200
>> >
>> > [Wed, 04 Aug 2010 22:57:52 GMT] [error] [emulator] Error in process
>> > <0.12957.0> with exit value:
>> {badarg,[{erlang,binary_to_term,[<<1885695232
>> >
>> >
>> bytes>>]},{couch_file,pread_term,2},{couch_db,make_doc,5},{couch_db,open_doc_int,3},{couch_view_updater,load_doc,5},{couch_view_updater,'-update/2-fun-2-',8},{couch_btree,stream_kv_node2,8...
>> >
>> > Anyone know where to start?
>> >
>> > I am running Ubuntu 10.04 server with couchdb 1.0 installed from source.
>> I
>> > have made sure the xulrunner settings are correct.
>> >
>> > Thanks,
>> >
>> > On Wed, Aug 4, 2010 at 12:18 PM, Gordon Leland Hempton
>> > <gh...@gmail.com>wrote:
>> >
>> > > I am working with a database of ~10 mil docs and 30GB size. I am
>> trying
>> > to
>> > > build a view consisting of a single field as the key and a null
>> document.
>> > >
>> > > After creating the view and trying to access it, the couchdb beam.smp
>> > > command eventually goes up to 12.3GB virtual memory and uses 3.7GB of
>> > > physical memory. Memory is clearly the bottleneck as the CPU usage is
>> > around
>> > > 5-15%. My system only has 4GB of memory so naturally all this paging
>> is
>> > > slowing everything down to a crawl.
>> > >
>> > > After a few minutes this usage will go back down, but it will get to
>> the
>> > > same levels every time I access the view (which is still building and
>> has
>> > > been for 12 hours).
>> > >
>> > > Is this normal? How can I improve this?
>> > >
>> > > --
>> > > Gordon L. Hempton
>> > > http://componentlab.com
>> > > http://hempton.com
>> > >
>> >
>> >
>> >
>> > --
>> > Gordon L. Hempton
>> > http://componentlab.com
>> > http://hempton.com
>> >
>>
>
>
>
> --
> Gordon L. Hempton
> http://componentlab.com
> http://hempton.com
>
--
Gordon L. Hempton
http://componentlab.com
http://hempton.com
Re: View Building Grinds System to Halt
Posted by Gordon Leland Hempton <gh...@gmail.com>.
Thanks for your responses!
The design document for this database is trivial. There is only the single
view which emits a single property as the key:
{
"_id": "_design/Trademark",
"_rev": "6-ecab7bf9158e188ab2c39d37c4a1fb4d",
"views": {
"by_filingDate": {
"map": "function(doc) { emit(doc.filingDate, null); }"
}
}
}
This exact same setup works on a smaller scale of data on our development boxes.
I have also ensured that ldconfig was run. I am actually using the
script provided on the wiki to update the xulrunner version.
Futhermore, before I fixed xulrunner, NONE of my views were working,
however now views are working fine on all databases exect ones of
large size.
Thanks,
Gordon
On Thu, Aug 5, 2010 at 1:25 PM, David Caylor <dc...@fh.org> wrote:
Did you run ldconfig after you set the xulrunner settings? I did that to
> myself last time I updated a Ubuntu box where I'd installed couchdb from
> source. I updated the xulrunner.conf, but forgot to run ldconfig, and got
> pretty much what you are describing.
>
> David Caylor
>
>
> On Wed, Aug 4, 2010 at 4:07 PM, Gordon Leland Hempton <ghempton@gmail.com
> >wrote:
>
> > After looking at the logs, it looks pretty bad:
> >
> > [Wed, 04 Aug 2010 22:57:28 GMT] [info] [<0.12883.0>] 127.0.0.1 - - 'GET'
> >
> >
> /trademark/_design/Trademark/_view/by_filingDate?limit=10&stale=ok&descending=true&include_docs=true
> > 200
> >
> > [Wed, 04 Aug 2010 22:57:52 GMT] [error] [emulator] Error in process
> > <0.12957.0> with exit value:
> {badarg,[{erlang,binary_to_term,[<<1885695232
> >
> >
> bytes>>]},{couch_file,pread_term,2},{couch_db,make_doc,5},{couch_db,open_doc_int,3},{couch_view_updater,load_doc,5},{couch_view_updater,'-update/2-fun-2-',8},{couch_btree,stream_kv_node2,8...
> >
> > Anyone know where to start?
> >
> > I am running Ubuntu 10.04 server with couchdb 1.0 installed from source.
> I
> > have made sure the xulrunner settings are correct.
> >
> > Thanks,
> >
> > On Wed, Aug 4, 2010 at 12:18 PM, Gordon Leland Hempton
> > <gh...@gmail.com>wrote:
> >
> > > I am working with a database of ~10 mil docs and 30GB size. I am trying
> > to
> > > build a view consisting of a single field as the key and a null
> document.
> > >
> > > After creating the view and trying to access it, the couchdb beam.smp
> > > command eventually goes up to 12.3GB virtual memory and uses 3.7GB of
> > > physical memory. Memory is clearly the bottleneck as the CPU usage is
> > around
> > > 5-15%. My system only has 4GB of memory so naturally all this paging is
> > > slowing everything down to a crawl.
> > >
> > > After a few minutes this usage will go back down, but it will get to
> the
> > > same levels every time I access the view (which is still building and
> has
> > > been for 12 hours).
> > >
> > > Is this normal? How can I improve this?
> > >
> > > --
> > > Gordon L. Hempton
> > > http://componentlab.com
> > > http://hempton.com
> > >
> >
> >
> >
> > --
> > Gordon L. Hempton
> > http://componentlab.com
> > http://hempton.com
> >
>
--
Gordon L. Hempton
http://componentlab.com
http://hempton.com
Re: View Building Grinds System to Halt
Posted by David Caylor <dc...@fh.org>.
Did you run ldconfig after you set the xulrunner settings? I did that to
myself last time I updated a Ubuntu box where I'd installed couchdb from
source. I updated the xulrunner.conf, but forgot to run ldconfig, and got
pretty much what you are describing.
David Caylor
On Wed, Aug 4, 2010 at 4:07 PM, Gordon Leland Hempton <gh...@gmail.com>wrote:
> After looking at the logs, it looks pretty bad:
>
> [Wed, 04 Aug 2010 22:57:28 GMT] [info] [<0.12883.0>] 127.0.0.1 - - 'GET'
>
> /trademark/_design/Trademark/_view/by_filingDate?limit=10&stale=ok&descending=true&include_docs=true
> 200
>
> [Wed, 04 Aug 2010 22:57:52 GMT] [error] [emulator] Error in process
> <0.12957.0> with exit value: {badarg,[{erlang,binary_to_term,[<<1885695232
>
> bytes>>]},{couch_file,pread_term,2},{couch_db,make_doc,5},{couch_db,open_doc_int,3},{couch_view_updater,load_doc,5},{couch_view_updater,'-update/2-fun-2-',8},{couch_btree,stream_kv_node2,8...
>
> Anyone know where to start?
>
> I am running Ubuntu 10.04 server with couchdb 1.0 installed from source. I
> have made sure the xulrunner settings are correct.
>
> Thanks,
>
> On Wed, Aug 4, 2010 at 12:18 PM, Gordon Leland Hempton
> <gh...@gmail.com>wrote:
>
> > I am working with a database of ~10 mil docs and 30GB size. I am trying
> to
> > build a view consisting of a single field as the key and a null document.
> >
> > After creating the view and trying to access it, the couchdb beam.smp
> > command eventually goes up to 12.3GB virtual memory and uses 3.7GB of
> > physical memory. Memory is clearly the bottleneck as the CPU usage is
> around
> > 5-15%. My system only has 4GB of memory so naturally all this paging is
> > slowing everything down to a crawl.
> >
> > After a few minutes this usage will go back down, but it will get to the
> > same levels every time I access the view (which is still building and has
> > been for 12 hours).
> >
> > Is this normal? How can I improve this?
> >
> > --
> > Gordon L. Hempton
> > http://componentlab.com
> > http://hempton.com
> >
>
>
>
> --
> Gordon L. Hempton
> http://componentlab.com
> http://hempton.com
>
Re: View Building Grinds System to Halt
Posted by Gordon Leland Hempton <gh...@gmail.com>.
After looking at the logs, it looks pretty bad:
[Wed, 04 Aug 2010 22:57:28 GMT] [info] [<0.12883.0>] 127.0.0.1 - - 'GET'
/trademark/_design/Trademark/_view/by_filingDate?limit=10&stale=ok&descending=true&include_docs=true
200
[Wed, 04 Aug 2010 22:57:52 GMT] [error] [emulator] Error in process
<0.12957.0> with exit value: {badarg,[{erlang,binary_to_term,[<<1885695232
bytes>>]},{couch_file,pread_term,2},{couch_db,make_doc,5},{couch_db,open_doc_int,3},{couch_view_updater,load_doc,5},{couch_view_updater,'-update/2-fun-2-',8},{couch_btree,stream_kv_node2,8...
Anyone know where to start?
I am running Ubuntu 10.04 server with couchdb 1.0 installed from source. I
have made sure the xulrunner settings are correct.
Thanks,
On Wed, Aug 4, 2010 at 12:18 PM, Gordon Leland Hempton
<gh...@gmail.com>wrote:
> I am working with a database of ~10 mil docs and 30GB size. I am trying to
> build a view consisting of a single field as the key and a null document.
>
> After creating the view and trying to access it, the couchdb beam.smp
> command eventually goes up to 12.3GB virtual memory and uses 3.7GB of
> physical memory. Memory is clearly the bottleneck as the CPU usage is around
> 5-15%. My system only has 4GB of memory so naturally all this paging is
> slowing everything down to a crawl.
>
> After a few minutes this usage will go back down, but it will get to the
> same levels every time I access the view (which is still building and has
> been for 12 hours).
>
> Is this normal? How can I improve this?
>
> --
> Gordon L. Hempton
> http://componentlab.com
> http://hempton.com
>
--
Gordon L. Hempton
http://componentlab.com
http://hempton.com