You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pylucene-dev@lucene.apache.org by Randomcoder <ra...@gmail.com> on 2014/06/18 05:02:50 UTC
pylucene stops with std::bad_alloc after index reaches ~3GB
Hi,
I've been using pylucene 4.8.0-1(the latest release) to index a lot of data(~100GB) on a machine
with 16GB of memory.
After the index reaches 3GB the following C++ run-time exception is being thrown:
terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted
I was able to recover a stacktrace with gdb(using the core dump file and the python binary) which
I'm attaching to this e-mail.
Has anyone bumped into this before ?
P.S. I've tried using jira to report this but I get "session timed out" when I'm trying
to create a new account.
Re: pylucene stops with std::bad_alloc after index reaches ~3GB
Posted by Andi Vajda <va...@apache.org>.
> On Jun 17, 2014, at 21:38, Randomcoder <ra...@gmail.com> wrote:
>
> On Wed, Jun 18, 2014 at 07:33:09AM +0300, Randomcoder wrote:
>>>> Thank you Andi, I will try to increase the jvm memory.
>>>> But if this was indeed a jvm problem, shouldn't a jvm exception be thrown
>>>> instead of the C++ runtime std::bad_alloc ?
>>>
>>> With JNI, all is possible. And without a stacktrace (no attachments please), it's just a guess.
>>
>> Ah, here is the stacktrace http://pastebin.com/MiMeyJxV
>
> I've pasted on the apache apaste.info service too http://apaste.info/cCj
It looks like it's crashing while making a java string from a python one.
Andi..
Re: pylucene stops with std::bad_alloc after index reaches ~3GB
Posted by Randomcoder <ra...@gmail.com>.
On Wed, Jun 18, 2014 at 07:33:09AM +0300, Randomcoder wrote:
> > > Thank you Andi, I will try to increase the jvm memory.
> > > But if this was indeed a jvm problem, shouldn't a jvm exception be thrown
> > > instead of the C++ runtime std::bad_alloc ?
> >
> > With JNI, all is possible. And without a stacktrace (no attachments please), it's just a guess.
>
> Ah, here is the stacktrace http://pastebin.com/MiMeyJxV
I've pasted on the apache apaste.info service too http://apaste.info/cCj
Re: pylucene stops with std::bad_alloc after index reaches ~3GB
Posted by Randomcoder <ra...@gmail.com>.
On Tue, Jun 17, 2014 at 09:25:27PM -0700, Andi Vajda wrote:
> > Thank you Andi, I will try to increase the jvm memory.
> > But if this was indeed a jvm problem, shouldn't a jvm exception be thrown
> > instead of the C++ runtime std::bad_alloc ?
>
> With JNI, all is possible. And without a stacktrace (no attachments please), it's just a guess.
Ah, here is the stacktrace http://pastebin.com/MiMeyJxV
Also, I've tried increasing the memory as you have suggested like this:
lucene.initVM(initialheap='5000m', maxheap='5001m', maxstack='1000m', vmargs=['-Djava.awt.headless=true'])
I've read about initialheap,maxheap and maxstack from
http://lucene.apache.org/pylucene/jcc/readme.html#api
The memory is still increasing at a steady rate. Now it's at 6.6% of 15718mb (1037mb)
>
> Andi..
Re: pylucene stops with std::bad_alloc after index reaches ~3GB
Posted by Andi Vajda <va...@apache.org>.
> On Jun 17, 2014, at 21:18, Randomcoder <ra...@gmail.com> wrote:
>
> On Tue, Jun 17, 2014 at 08:54:50PM -0700, Andi Vajda wrote:
>>> [..]
>>> terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted
>>> [..]
>>> Has anyone bumped into this before ?
>>
>> You might need to increase the RAM default for the JVM when calling
>> initVM(). Please, see initVM() docs on this page for more details:
>> http://lucene.apache.org/pylucene/jcc/features.html
>
> Thank you Andi, I will try to increase the jvm memory.
> But if this was indeed a jvm problem, shouldn't a jvm exception be thrown
> instead of the C++ runtime std::bad_alloc ?
With JNI, all is possible. And without a stacktrace (no attachments please), it's just a guess.
Andi..
>
>>
>> Andi..
Re: pylucene stops with std::bad_alloc after index reaches ~3GB
Posted by Randomcoder <ra...@gmail.com>.
On Tue, Jun 17, 2014 at 08:54:50PM -0700, Andi Vajda wrote:
> > [..]
> > terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted
> > [..]
> >Has anyone bumped into this before ?
>
> You might need to increase the RAM default for the JVM when calling
> initVM(). Please, see initVM() docs on this page for more details:
> http://lucene.apache.org/pylucene/jcc/features.html
Thank you Andi, I will try to increase the jvm memory.
But if this was indeed a jvm problem, shouldn't a jvm exception be thrown
instead of the C++ runtime std::bad_alloc ?
>
> Andi..
Re: pylucene stops with std::bad_alloc after index reaches ~3GB
Posted by Andi Vajda <va...@apache.org>.
On Wed, 18 Jun 2014, Randomcoder wrote:
>
> Hi,
>
> I've been using pylucene 4.8.0-1(the latest release) to index a lot of data(~100GB) on a machine
> with 16GB of memory.
> After the index reaches 3GB the following C++ run-time exception is being thrown:
>
> terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted
>
> I was able to recover a stacktrace with gdb(using the core dump file and the python binary) which
> I'm attaching to this e-mail.
>
> Has anyone bumped into this before ?
You might need to increase the RAM default for the JVM when calling
initVM(). Please, see initVM() docs on this page for more details:
http://lucene.apache.org/pylucene/jcc/features.html
Andi..
>
>
> P.S. I've tried using jira to report this but I get "session timed out" when I'm trying
> to create a new account.
>