You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Alexander Quick <al...@alexquick.com> on 2009/10/23 18:55:14 UTC
fails to allocate heap on bulk save
Hi,
I'm having an issue where CouchDB crashes consistently while bulk
saving(all_or_nothing is off). It completes fewer than a million
documents of the following form before choking:
{
"_id": "an_id",
"user_id": "a_user",
"attribute_one":"<7chars",
"attribute_two":"<7chars",
"attribute_three":"<7chars",
}
CouchDB 0.10 is running on Windows Server 2008 (might be my first
mistake) and trying to insert only in 3k batches, so I don't think
individual batch size is an issue(along with the fact that the first
30-40 batches succeed).
I get the following error:
Crash dump was written to: erl_crash.dump
eheap_alloc: Cannot allocate 50152200 bytes of memory (of type "heap").
Sometimes it fails while trying to allocate some other type of memory,
but heap is most common.
The machine still has plenty of memory left when this happens, and
we're talking about a 50mb allocation. Even with the vm heap parameter
set to 1gb, it chokes around 700mb-- either way this shouldn't be an
issue.
The only thing I can think of that separates this from an entirely run-
of-the-mill use case is the fact that I'm inserting into approximately
100 databases. Though not concurrently, but rather say a batch to the
first then a batch to the second after the first completes.
Nonetheless, the vm keeps piling on new processes for some reason.
The dump is available here: http://sandbox.stoopdev.com/fail/erl_crash.dump
Please let me know if you have any ideas.
Thanks in advance,
Alex Quick
Re: fails to allocate heap on bulk save
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Oct 23, 2009 at 1:30 PM, Alexander Quick <al...@alexquick.com> wrote:
> Paul,
>
> Yeah, memory usage just grows and grows.
> Do you think this might just be a problem with the windows version of
> CouchDB or Erlang?
> I couldn't find anything online about a memory leak in this sort of case for
> either project.
>
> Alex
Well, its definitely not par for CouchDB so I'd guess its something
specific to your setup. Its possible (thought unlikely) that Erlang or
CouchDB does something weird on Windows.
My random shot in the dark guess: Is your client leaking open socket
connections? It could be possible that if you leave the connection
open the Erlang PID just sits idle without being garbage collected,
then as you open more and more sockets you hit a the memory limit. The
solution would be to make sure to close the socket or make sure to
reuse it for the following requests. Either of those should make sure
that it gets garbage collected.
Paul Davis
>
> On Oct 23, 2009, at 1:20 PM, Paul Davis wrote:
>
>> On Fri, Oct 23, 2009 at 12:55 PM, Alexander Quick <al...@alexquick.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I'm having an issue where CouchDB crashes consistently while bulk
>>> saving(all_or_nothing is off). It completes fewer than a million
>>> documents
>>> of the following form before choking:
>>> {
>>> "_id": "an_id",
>>> "user_id": "a_user",
>>> "attribute_one":"<7chars",
>>> "attribute_two":"<7chars",
>>> "attribute_three":"<7chars",
>>> }
>>>
>>> CouchDB 0.10 is running on Windows Server 2008 (might be my first
>>> mistake)
>>> and trying to insert only in 3k batches, so I don't think individual
>>> batch
>>> size is an issue(along with the fact that the first 30-40 batches
>>> succeed).
>>>
>>> I get the following error:
>>> Crash dump was written to: erl_crash.dump
>>> eheap_alloc: Cannot allocate 50152200 bytes of memory (of type "heap").
>>> Sometimes it fails while trying to allocate some other type of memory,
>>> but
>>> heap is most common.
>>>
>>> The machine still has plenty of memory left when this happens, and we're
>>> talking about a 50mb allocation. Even with the vm heap parameter set to
>>> 1gb,
>>> it chokes around 700mb-- either way this shouldn't be an issue.
>>>
>>> The only thing I can think of that separates this from an entirely
>>> run-of-the-mill use case is the fact that I'm inserting into
>>> approximately
>>> 100 databases. Though not concurrently, but rather say a batch to the
>>> first
>>> then a batch to the second after the first completes. Nonetheless, the vm
>>> keeps piling on new processes for some reason.
>>>
>>> The dump is available here:
>>> http://sandbox.stoopdev.com/fail/erl_crash.dump
>>> Please let me know if you have any ideas.
>>>
>>> Thanks in advance,
>>> Alex Quick
>>>
>>
>> Alex,
>>
>> Most odd. Can you see what the memory profile does as you do your
>> bulk_docs calls? Does it appear to just continuously grow and not
>> release RAM as you make multiple calls? It shouldn't hold onto any
>> memory after a _bulk_docs request is completed.
>>
>> Paul Davis
>
>
Re: fails to allocate heap on bulk save
Posted by Alexander Quick <al...@alexquick.com>.
Paul,
Yeah, memory usage just grows and grows.
Do you think this might just be a problem with the windows version of
CouchDB or Erlang?
I couldn't find anything online about a memory leak in this sort of
case for either project.
Alex
On Oct 23, 2009, at 1:20 PM, Paul Davis wrote:
> On Fri, Oct 23, 2009 at 12:55 PM, Alexander Quick
> <al...@alexquick.com> wrote:
>> Hi,
>>
>> I'm having an issue where CouchDB crashes consistently while bulk
>> saving(all_or_nothing is off). It completes fewer than a million
>> documents
>> of the following form before choking:
>> {
>> "_id": "an_id",
>> "user_id": "a_user",
>> "attribute_one":"<7chars",
>> "attribute_two":"<7chars",
>> "attribute_three":"<7chars",
>> }
>>
>> CouchDB 0.10 is running on Windows Server 2008 (might be my first
>> mistake)
>> and trying to insert only in 3k batches, so I don't think
>> individual batch
>> size is an issue(along with the fact that the first 30-40 batches
>> succeed).
>>
>> I get the following error:
>> Crash dump was written to: erl_crash.dump
>> eheap_alloc: Cannot allocate 50152200 bytes of memory (of type
>> "heap").
>> Sometimes it fails while trying to allocate some other type of
>> memory, but
>> heap is most common.
>>
>> The machine still has plenty of memory left when this happens, and
>> we're
>> talking about a 50mb allocation. Even with the vm heap parameter
>> set to 1gb,
>> it chokes around 700mb-- either way this shouldn't be an issue.
>>
>> The only thing I can think of that separates this from an entirely
>> run-of-the-mill use case is the fact that I'm inserting into
>> approximately
>> 100 databases. Though not concurrently, but rather say a batch to
>> the first
>> then a batch to the second after the first completes. Nonetheless,
>> the vm
>> keeps piling on new processes for some reason.
>>
>> The dump is available here: http://sandbox.stoopdev.com/fail/erl_crash.dump
>> Please let me know if you have any ideas.
>>
>> Thanks in advance,
>> Alex Quick
>>
>
> Alex,
>
> Most odd. Can you see what the memory profile does as you do your
> bulk_docs calls? Does it appear to just continuously grow and not
> release RAM as you make multiple calls? It shouldn't hold onto any
> memory after a _bulk_docs request is completed.
>
> Paul Davis
Re: fails to allocate heap on bulk save
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Oct 23, 2009 at 12:55 PM, Alexander Quick <al...@alexquick.com> wrote:
> Hi,
>
> I'm having an issue where CouchDB crashes consistently while bulk
> saving(all_or_nothing is off). It completes fewer than a million documents
> of the following form before choking:
> {
> "_id": "an_id",
> "user_id": "a_user",
> "attribute_one":"<7chars",
> "attribute_two":"<7chars",
> "attribute_three":"<7chars",
> }
>
> CouchDB 0.10 is running on Windows Server 2008 (might be my first mistake)
> and trying to insert only in 3k batches, so I don't think individual batch
> size is an issue(along with the fact that the first 30-40 batches succeed).
>
> I get the following error:
> Crash dump was written to: erl_crash.dump
> eheap_alloc: Cannot allocate 50152200 bytes of memory (of type "heap").
> Sometimes it fails while trying to allocate some other type of memory, but
> heap is most common.
>
> The machine still has plenty of memory left when this happens, and we're
> talking about a 50mb allocation. Even with the vm heap parameter set to 1gb,
> it chokes around 700mb-- either way this shouldn't be an issue.
>
> The only thing I can think of that separates this from an entirely
> run-of-the-mill use case is the fact that I'm inserting into approximately
> 100 databases. Though not concurrently, but rather say a batch to the first
> then a batch to the second after the first completes. Nonetheless, the vm
> keeps piling on new processes for some reason.
>
> The dump is available here: http://sandbox.stoopdev.com/fail/erl_crash.dump
> Please let me know if you have any ideas.
>
> Thanks in advance,
> Alex Quick
>
Alex,
Most odd. Can you see what the memory profile does as you do your
bulk_docs calls? Does it appear to just continuously grow and not
release RAM as you make multiple calls? It shouldn't hold onto any
memory after a _bulk_docs request is completed.
Paul Davis