You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ameya Aware <am...@gmail.com> on 2014/07/24 15:32:12 UTC
Java heap space error
Hi
I am in process of indexing around 2,00,000 documents.
I have increase java jeap space to 4 GB using below command :
java -Xmx4096M -Xms4096M -jar start.jar
Still after indexing around 15000 documents it gives java heap space error
again.
Any fix for this?
Thanks,
Ameya
RE: Java heap space error
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
Steve Rowe [sarowe@gmail.com] wrote:
> 1 Lakh (aka Lac) = 10^5 is written as 1,00,000
>
> It’s used in Bangladesh, India, Myanmar, Nepal, Pakistan, and Sri Lanka, roughly 1/4 of the world’s population.
Yet still it causes confusion and distracts from the issue. Let's just stick to metric, okay?
- Toke Eskildsen
Re: Java heap space error
Posted by Steve Rowe <sa...@gmail.com>.
On Jul 25, 2014, at 9:13 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 7/24/2014 7:53 AM, Ameya Aware wrote:
> The odd location of the commas in the start of this thread make it hard
> to understand exactly what numbers you were trying to say
On Jul 24, 2014, at 9:32 AM, Ameya Aware <am...@gmail.com> wrote:
> I am in process of indexing around 2,00,000 documents.
1 Lakh (aka Lac) = 10^5 is written as 1,00,000
It’s used in Bangladesh, India, Myanmar, Nepal, Pakistan, and Sri Lanka, roughly 1/4 of the world’s population.
<http://en.wikipedia.org/wiki/Lakh>
Re: Java heap space error
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/24/2014 7:53 AM, Ameya Aware wrote:
> I did not make any other change than this.. rest of the settings are
> default.
>
> Do i need to set garbage collection strategy?
The collector chosen and its and tuning params can have a massive impact
on performance, but it will make no difference at all if you are getting
OutOfMemoryError exceptions. This means the program is trying to
allocate more memory than it has been told it can allocate. Changing
the garbage collector will not change Java's response when the program
wants to allocate too much memory.
The odd location of the commas in the start of this thread make it hard
to understand exactly what numbers you were trying to say, but I think
you were saying that you were trying to index 200000 documents and it
died after indexing 15000.
How big was the solr index before you started indexing, both in number
of documents and disk space consumed? How are you doing the indexing?
Is it being done with requests to the /update handler, or are you using
the dataimport handler to import from somewhere, like a database?
Is it a single index, or distributed? Are you running in "normal" mode
or SolrCloud? Can you share your solrconfig.xml file so we can look for
possible problems?
I already gave you a wiki URL that gives possible reasons for needing a
very large heap, and some things you can do to reduce the requirements.
Thanks,
Shawn
Re: Java heap space error
Posted by Boon Low <bo...@dctfh.com>.
How about simply increasing the heap size if RAM is available? You should also check the update handler config, e.g. auto commit, if docs aren’t being written to disk, they would be hanging around in memory. And “openSearcher” setting too as opening new searchers consumes memory, especially if expensive warm-up requests are configured.
Setup some graphs, monitoring the JVM heap, indexing rate, pending docs etc etc.
Boon
---
Boon Low
Lead Big Data / Search Engineer
DCT Family History
On 24 Jul 2014, at 15:12, François Schiettecatte <fs...@gmail.com>> wrote:
A default garbage collector will be chosen for you by the VM, might help to get the stack trace to look at.
François
On Jul 24, 2014, at 10:06 AM, Ameya Aware <am...@gmail.com>> wrote:
ooh ok.
So you want to say that since i am using large heap but didnt set my
garbage collection, thats why i why getting java heap space error?
On Thu, Jul 24, 2014 at 9:58 AM, Marcello Lorenzi <ml...@sorint.it>>
wrote:
I think that on large heap is suggested to monitor the garbage collection
behavior and try to add a strategy adapted to your performance. On my
production environment with a heap of 6 GB I set this parameter (server
with 8 cores):
-server -Xms6144m -Xmx6144m -XX:MaxPermSize=512m
-Dcom.sun.management.jmxremote -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode -XX:+CMSParallelRemarkEnabled
-XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70
-XX:ConcGCThreads=6 -XX:ParallelGCThreads=6
Marcello
On 07/24/2014 03:53 PM, Ameya Aware wrote:
I did not make any other change than this.. rest of the settings are
default.
Do i need to set garbage collection strategy?
On Thu, Jul 24, 2014 at 9:49 AM, Marcello Lorenzi <ml...@sorint.it>>
wrote:
Hi,
Did you set a Garbage collection strategy on your JVM ?
Marcello
On 07/24/2014 03:32 PM, Ameya Aware wrote:
Hi
I am in process of indexing around 2,00,000 documents.
I have increase java jeap space to 4 GB using below command :
java -Xmx4096M -Xms4096M -jar start.jar
Still after indexing around 15000 documents it gives java heap space
error
again.
Any fix for this?
Thanks,
Ameya
________________________________
This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of D.C. Thomson Family History shall be understood as neither given nor endorsed by it.
________________________________
This email has been checked for virus and other malicious content prior to
leaving our network.
Re: Java heap space error
Posted by François Schiettecatte <fs...@gmail.com>.
A default garbage collector will be chosen for you by the VM, might help to get the stack trace to look at.
François
On Jul 24, 2014, at 10:06 AM, Ameya Aware <am...@gmail.com> wrote:
> ooh ok.
>
> So you want to say that since i am using large heap but didnt set my
> garbage collection, thats why i why getting java heap space error?
>
>
>
>
>
> On Thu, Jul 24, 2014 at 9:58 AM, Marcello Lorenzi <ml...@sorint.it>
> wrote:
>
>> I think that on large heap is suggested to monitor the garbage collection
>> behavior and try to add a strategy adapted to your performance. On my
>> production environment with a heap of 6 GB I set this parameter (server
>> with 8 cores):
>>
>> -server -Xms6144m -Xmx6144m -XX:MaxPermSize=512m
>> -Dcom.sun.management.jmxremote -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
>> -XX:+CMSIncrementalMode -XX:+CMSParallelRemarkEnabled
>> -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70
>> -XX:ConcGCThreads=6 -XX:ParallelGCThreads=6
>>
>> Marcello
>>
>>
>> On 07/24/2014 03:53 PM, Ameya Aware wrote:
>>
>> I did not make any other change than this.. rest of the settings are
>> default.
>>
>> Do i need to set garbage collection strategy?
>>
>>
>> On Thu, Jul 24, 2014 at 9:49 AM, Marcello Lorenzi <ml...@sorint.it>
>> wrote:
>>
>>> Hi,
>>> Did you set a Garbage collection strategy on your JVM ?
>>>
>>> Marcello
>>>
>>>
>>> On 07/24/2014 03:32 PM, Ameya Aware wrote:
>>>
>>>> Hi
>>>>
>>>> I am in process of indexing around 2,00,000 documents.
>>>>
>>>> I have increase java jeap space to 4 GB using below command :
>>>>
>>>> java -Xmx4096M -Xms4096M -jar start.jar
>>>>
>>>> Still after indexing around 15000 documents it gives java heap space
>>>> error
>>>> again.
>>>>
>>>>
>>>> Any fix for this?
>>>>
>>>> Thanks,
>>>> Ameya
>>>>
>>>>
>>>
>>
>>
Re: Java heap space error
Posted by Ameya Aware <am...@gmail.com>.
ooh ok.
So you want to say that since i am using large heap but didnt set my
garbage collection, thats why i why getting java heap space error?
On Thu, Jul 24, 2014 at 9:58 AM, Marcello Lorenzi <ml...@sorint.it>
wrote:
> I think that on large heap is suggested to monitor the garbage collection
> behavior and try to add a strategy adapted to your performance. On my
> production environment with a heap of 6 GB I set this parameter (server
> with 8 cores):
>
> -server -Xms6144m -Xmx6144m -XX:MaxPermSize=512m
> -Dcom.sun.management.jmxremote -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:+CMSIncrementalMode -XX:+CMSParallelRemarkEnabled
> -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70
> -XX:ConcGCThreads=6 -XX:ParallelGCThreads=6
>
> Marcello
>
>
> On 07/24/2014 03:53 PM, Ameya Aware wrote:
>
> I did not make any other change than this.. rest of the settings are
> default.
>
> Do i need to set garbage collection strategy?
>
>
> On Thu, Jul 24, 2014 at 9:49 AM, Marcello Lorenzi <ml...@sorint.it>
> wrote:
>
>> Hi,
>> Did you set a Garbage collection strategy on your JVM ?
>>
>> Marcello
>>
>>
>> On 07/24/2014 03:32 PM, Ameya Aware wrote:
>>
>>> Hi
>>>
>>> I am in process of indexing around 2,00,000 documents.
>>>
>>> I have increase java jeap space to 4 GB using below command :
>>>
>>> java -Xmx4096M -Xms4096M -jar start.jar
>>>
>>> Still after indexing around 15000 documents it gives java heap space
>>> error
>>> again.
>>>
>>>
>>> Any fix for this?
>>>
>>> Thanks,
>>> Ameya
>>>
>>>
>>
>
>
Re: Java heap space error
Posted by Marcello Lorenzi <ml...@sorint.it>.
I think that on large heap is suggested to monitor the garbage
collection behavior and try to add a strategy adapted to your
performance. On my production environment with a heap of 6 GB I set
this parameter (server with 8 cores):
-server -Xms6144m -Xmx6144m -XX:MaxPermSize=512m
-Dcom.sun.management.jmxremote -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode -XX:+CMSParallelRemarkEnabled
-XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70
-XX:ConcGCThreads=6 -XX:ParallelGCThreads=6
Marcello
On 07/24/2014 03:53 PM, Ameya Aware wrote:
> I did not make any other change than this.. rest of the settings are
> default.
>
> Do i need to set garbage collection strategy?
>
>
> On Thu, Jul 24, 2014 at 9:49 AM, Marcello Lorenzi <mlorenzi@sorint.it
> <ma...@sorint.it>> wrote:
>
> Hi,
> Did you set a Garbage collection strategy on your JVM ?
>
> Marcello
>
>
> On 07/24/2014 03:32 PM, Ameya Aware wrote:
>
> Hi
>
> I am in process of indexing around 2,00,000 documents.
>
> I have increase java jeap space to 4 GB using below command :
>
> java -Xmx4096M -Xms4096M -jar start.jar
>
> Still after indexing around 15000 documents it gives java heap
> space error
> again.
>
>
> Any fix for this?
>
> Thanks,
> Ameya
>
>
>
Re: Java heap space error
Posted by Ameya Aware <am...@gmail.com>.
I did not make any other change than this.. rest of the settings are
default.
Do i need to set garbage collection strategy?
On Thu, Jul 24, 2014 at 9:49 AM, Marcello Lorenzi <ml...@sorint.it>
wrote:
> Hi,
> Did you set a Garbage collection strategy on your JVM ?
>
> Marcello
>
>
> On 07/24/2014 03:32 PM, Ameya Aware wrote:
>
>> Hi
>>
>> I am in process of indexing around 2,00,000 documents.
>>
>> I have increase java jeap space to 4 GB using below command :
>>
>> java -Xmx4096M -Xms4096M -jar start.jar
>>
>> Still after indexing around 15000 documents it gives java heap space error
>> again.
>>
>>
>> Any fix for this?
>>
>> Thanks,
>> Ameya
>>
>>
>
Re: Java heap space error
Posted by Marcello Lorenzi <ml...@sorint.it>.
Hi,
Did you set a Garbage collection strategy on your JVM ?
Marcello
On 07/24/2014 03:32 PM, Ameya Aware wrote:
> Hi
>
> I am in process of indexing around 2,00,000 documents.
>
> I have increase java jeap space to 4 GB using below command :
>
> java -Xmx4096M -Xms4096M -jar start.jar
>
> Still after indexing around 15000 documents it gives java heap space error
> again.
>
>
> Any fix for this?
>
> Thanks,
> Ameya
>