You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by John Evans <jo...@jpevans.com> on 2008/07/04 07:37:06 UTC
Erlang crash on memory allocation attempt
Hi,
I'm really eager to write up a great couchdb success story about how my
stealth mode startup has used couchdb to great effect.... but I can't yet,
since there is at least one show stopper for me -- specifically, this:
[info] [<0.84.0>] 127.0.0.1 - - "GET /products/_design%2Fproduct_with_offer"
304
Crash dump was written to: erl_crash.dump
eheap_alloc: Cannot allocate 2280657000 bytes of memory (of type "heap").
Aborted
Which I get when it tries to update my view, which looks like this
(cut/pasted from Futon):
{
"all": {
"map": "\n\t\t\t\tfunction(doc) { \n\t\t\t\t\tif ('Product' ==
doc.type) {\n\t\t\t\t\t\temit([doc._id, 0], doc)\n\t\t\t\t\t} else if
('Offer' == doc.type) {\n\t\t\t\t\t\temit([doc.product, 1],
doc)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t"
}
}
Here it is again from the source code I use to create it (which uses my home
grown client library but should be easy to understand and is more readable):
create_or_replace_view(database, "product_with_offer", {
"all": {
"map": """
function(doc) {
if ('Product' == doc.type) {
emit([doc._id, 0], doc)
} else if ('Offer' == doc.type) {
emit([doc.product, 1], doc)
}
}
"""
}
})
The database has 12,492 documents and is compacted to a size of 85.4m.
The machine this is running on only has 2 gigs of memory and 1 gig of swap
with over 1gig of other stuff running all the time, so obviously the basic
problem is that couchdb is trying to allocate more memory than is available
and one solution would be to add RAM to the box, but it really seems to me
as if generating a view on a dataset that is 85.4 megs should not require
over 2 gigs of RAM. Am I doing something stupid in my view code? I am out
of ideas on what to investigate next. Any ideas?
Thanks,
-
John
P.S. There are a few other non show-stopper issues that I've run into with
couchdb (e.g. the document counts in Futon being wrong in various
situations) but other than the show stopper above, it looks as if it
would/will be of great value to our product development effort.... so much
so in fact, that thanks to some ugly workarounds (untenable in the long run,
but doing the trick in the short run) even with this "show stopper" we are
already using it very successfully to accelerate our development... and we
have just barely scratched the surface of it's capabilities I am sure. So
you can consider this a minor success story already. :)
Re: Erlang crash on memory allocation attempt
Posted by John Evans <jo...@jpevans.com>.
Turns out I was wrong about which environment the error was occurring in .
It was actually happening in our Linux environment where the Erlang version
was out of date. Upgrading to R12B did in fact fix the problem.
Thanks.
-
John
On Fri, Jul 4, 2008 at 10:13 AM, John Evans <jo...@jpevans.com> wrote:
> daedalus:~ john$ port installed erlang
> The following ports are currently installed:
> erlang @R12B-0_0 (active)
>
> Here is the startup banner in case that helps:
>
> Erlang (BEAM) emulator version 5.6 [source] [async-threads:0]
> [kernel-poll:false]
>
> Which reminds me, this is on a PPC Mac (running Leopard) in case that
> matters.
>
> -
> John
>
>
> On Fri, Jul 4, 2008 at 4:46 AM, Damien Katz <da...@gmail.com> wrote:
>
>> What version of Erlang are you running? Make re you are running the latest
>> (R12B).
>>
>> -Damien
>>
>>
>> On Jul 4, 2008, at 1:37 AM, John Evans wrote:
>>
>> Hi,
>>>
>>> I'm really eager to write up a great couchdb success story about how my
>>> stealth mode startup has used couchdb to great effect.... but I can't
>>> yet,
>>> since there is at least one show stopper for me -- specifically, this:
>>>
>>> [info] [<0.84.0>] 127.0.0.1 - - "GET
>>> /products/_design%2Fproduct_with_offer"
>>> 304
>>>
>>> Crash dump was written to: erl_crash.dump
>>> eheap_alloc: Cannot allocate 2280657000 bytes of memory (of type "heap").
>>> Aborted
>>>
>>> Which I get when it tries to update my view, which looks like this
>>> (cut/pasted from Futon):
>>>
>>> {
>>> "all": {
>>> "map": "\n\t\t\t\tfunction(doc) { \n\t\t\t\t\tif ('Product' ==
>>> doc.type) {\n\t\t\t\t\t\temit([doc._id, 0], doc)\n\t\t\t\t\t} else if
>>> ('Offer' == doc.type) {\n\t\t\t\t\t\temit([doc.product, 1],
>>> doc)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t"
>>> }
>>> }
>>>
>>> Here it is again from the source code I use to create it (which uses my
>>> home
>>> grown client library but should be easy to understand and is more
>>> readable):
>>>
>>> create_or_replace_view(database, "product_with_offer", {
>>> "all": {
>>> "map": """
>>> function(doc) {
>>> if ('Product' == doc.type) {
>>> emit([doc._id, 0], doc)
>>> } else if ('Offer' == doc.type) {
>>> emit([doc.product, 1], doc)
>>> }
>>> }
>>> """
>>> }
>>> })
>>>
>>> The database has 12,492 documents and is compacted to a size of 85.4m.
>>>
>>> The machine this is running on only has 2 gigs of memory and 1 gig of
>>> swap
>>> with over 1gig of other stuff running all the time, so obviously the
>>> basic
>>> problem is that couchdb is trying to allocate more memory than is
>>> available
>>> and one solution would be to add RAM to the box, but it really seems to
>>> me
>>> as if generating a view on a dataset that is 85.4 megs should not require
>>> over 2 gigs of RAM. Am I doing something stupid in my view code? I am
>>> out
>>> of ideas on what to investigate next. Any ideas?
>>>
>>> Thanks,
>>> -
>>> John
>>>
>>> P.S. There are a few other non show-stopper issues that I've run into
>>> with
>>> couchdb (e.g. the document counts in Futon being wrong in various
>>> situations) but other than the show stopper above, it looks as if it
>>> would/will be of great value to our product development effort.... so
>>> much
>>> so in fact, that thanks to some ugly workarounds (untenable in the long
>>> run,
>>> but doing the trick in the short run) even with this "show stopper" we
>>> are
>>> already using it very successfully to accelerate our development... and
>>> we
>>> have just barely scratched the surface of it's capabilities I am sure.
>>> So
>>> you can consider this a minor success story already. :)
>>>
>>
>>
>
Re: Erlang crash on memory allocation attempt
Posted by John Evans <jo...@jpevans.com>.
daedalus:~ john$ port installed erlang
The following ports are currently installed:
erlang @R12B-0_0 (active)
Here is the startup banner in case that helps:
Erlang (BEAM) emulator version 5.6 [source] [async-threads:0]
[kernel-poll:false]
Which reminds me, this is on a PPC Mac (running Leopard) in case that
matters.
-
John
On Fri, Jul 4, 2008 at 4:46 AM, Damien Katz <da...@gmail.com> wrote:
> What version of Erlang are you running? Make re you are running the latest
> (R12B).
>
> -Damien
>
>
> On Jul 4, 2008, at 1:37 AM, John Evans wrote:
>
> Hi,
>>
>> I'm really eager to write up a great couchdb success story about how my
>> stealth mode startup has used couchdb to great effect.... but I can't yet,
>> since there is at least one show stopper for me -- specifically, this:
>>
>> [info] [<0.84.0>] 127.0.0.1 - - "GET
>> /products/_design%2Fproduct_with_offer"
>> 304
>>
>> Crash dump was written to: erl_crash.dump
>> eheap_alloc: Cannot allocate 2280657000 bytes of memory (of type "heap").
>> Aborted
>>
>> Which I get when it tries to update my view, which looks like this
>> (cut/pasted from Futon):
>>
>> {
>> "all": {
>> "map": "\n\t\t\t\tfunction(doc) { \n\t\t\t\t\tif ('Product' ==
>> doc.type) {\n\t\t\t\t\t\temit([doc._id, 0], doc)\n\t\t\t\t\t} else if
>> ('Offer' == doc.type) {\n\t\t\t\t\t\temit([doc.product, 1],
>> doc)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t"
>> }
>> }
>>
>> Here it is again from the source code I use to create it (which uses my
>> home
>> grown client library but should be easy to understand and is more
>> readable):
>>
>> create_or_replace_view(database, "product_with_offer", {
>> "all": {
>> "map": """
>> function(doc) {
>> if ('Product' == doc.type) {
>> emit([doc._id, 0], doc)
>> } else if ('Offer' == doc.type) {
>> emit([doc.product, 1], doc)
>> }
>> }
>> """
>> }
>> })
>>
>> The database has 12,492 documents and is compacted to a size of 85.4m.
>>
>> The machine this is running on only has 2 gigs of memory and 1 gig of swap
>> with over 1gig of other stuff running all the time, so obviously the basic
>> problem is that couchdb is trying to allocate more memory than is
>> available
>> and one solution would be to add RAM to the box, but it really seems to me
>> as if generating a view on a dataset that is 85.4 megs should not require
>> over 2 gigs of RAM. Am I doing something stupid in my view code? I am
>> out
>> of ideas on what to investigate next. Any ideas?
>>
>> Thanks,
>> -
>> John
>>
>> P.S. There are a few other non show-stopper issues that I've run into with
>> couchdb (e.g. the document counts in Futon being wrong in various
>> situations) but other than the show stopper above, it looks as if it
>> would/will be of great value to our product development effort.... so much
>> so in fact, that thanks to some ugly workarounds (untenable in the long
>> run,
>> but doing the trick in the short run) even with this "show stopper" we are
>> already using it very successfully to accelerate our development... and we
>> have just barely scratched the surface of it's capabilities I am sure. So
>> you can consider this a minor success story already. :)
>>
>
>
Re: Erlang crash on memory allocation attempt
Posted by Damien Katz <da...@gmail.com>.
What version of Erlang are you running? Make re you are running the
latest (R12B).
-Damien
On Jul 4, 2008, at 1:37 AM, John Evans wrote:
> Hi,
>
> I'm really eager to write up a great couchdb success story about how
> my
> stealth mode startup has used couchdb to great effect.... but I
> can't yet,
> since there is at least one show stopper for me -- specifically, this:
>
> [info] [<0.84.0>] 127.0.0.1 - - "GET /products/_design
> %2Fproduct_with_offer"
> 304
>
> Crash dump was written to: erl_crash.dump
> eheap_alloc: Cannot allocate 2280657000 bytes of memory (of type
> "heap").
> Aborted
>
> Which I get when it tries to update my view, which looks like this
> (cut/pasted from Futon):
>
> {
> "all": {
> "map": "\n\t\t\t\tfunction(doc) { \n\t\t\t\t\tif ('Product' ==
> doc.type) {\n\t\t\t\t\t\temit([doc._id, 0], doc)\n\t\t\t\t\t} else if
> ('Offer' == doc.type) {\n\t\t\t\t\t\temit([doc.product, 1],
> doc)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t"
> }
> }
>
> Here it is again from the source code I use to create it (which uses
> my home
> grown client library but should be easy to understand and is more
> readable):
>
> create_or_replace_view(database, "product_with_offer", {
> "all": {
> "map": """
> function(doc) {
> if ('Product' == doc.type) {
> emit([doc._id, 0], doc)
> } else if ('Offer' == doc.type) {
> emit([doc.product, 1], doc)
> }
> }
> """
> }
> })
>
> The database has 12,492 documents and is compacted to a size of 85.4m.
>
> The machine this is running on only has 2 gigs of memory and 1 gig
> of swap
> with over 1gig of other stuff running all the time, so obviously the
> basic
> problem is that couchdb is trying to allocate more memory than is
> available
> and one solution would be to add RAM to the box, but it really seems
> to me
> as if generating a view on a dataset that is 85.4 megs should not
> require
> over 2 gigs of RAM. Am I doing something stupid in my view code? I
> am out
> of ideas on what to investigate next. Any ideas?
>
> Thanks,
> -
> John
>
> P.S. There are a few other non show-stopper issues that I've run
> into with
> couchdb (e.g. the document counts in Futon being wrong in various
> situations) but other than the show stopper above, it looks as if it
> would/will be of great value to our product development effort....
> so much
> so in fact, that thanks to some ugly workarounds (untenable in the
> long run,
> but doing the trick in the short run) even with this "show stopper"
> we are
> already using it very successfully to accelerate our development...
> and we
> have just barely scratched the surface of it's capabilities I am
> sure. So
> you can consider this a minor success story already. :)