You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by 308181687 <30...@qq.com> on 2014/09/13 10:25:01 UTC
OutOfMemoryError throwed by SimpleMergedSegmentWarmer
Hi, all
we got an OutOfMemoryError throwed by SimpleMergedSegmentWarmer. We use lucene 4.7， and access index file by NRTCachingDirectory/MMapDirectory. Could any body give me a hand? Strack trace is as follows:
org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:228)
at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:188)
at org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:166)
at org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:519)
at org.apache.lucene.index.SimpleMergedSegmentWarmer.warm(SimpleMergedSegmentWarmer.java:52)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4275)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3743)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
Thanks & Best Regards!
Re: OutOfMemoryError throwed by SimpleMergedSegmentWarmer
Posted by 308181687 <30...@qq.com>.
Hi, mike
Is there any config option which let IndexReader to omit all norms and not to load norms to JVM heap?
Thanks & Best Regards!
------------------ Original ------------------
From: "lubin";<30...@qq.com>;
Date: Sat, Sep 13, 2014 08:20 PM
To: "java-user"<ja...@lucene.apache.org>;
Subject: Re: OutOfMemoryError throwed by SimpleMergedSegmentWarmer
Hi, Mike
In our use case, we have thousands of index fields, different kind of document have different fields. Do you meant that norms field will consume large memory? Why?
If we decide to disable norms, do we need to rebuild our index entirely? By the way, We have 8 million documents and our jvm heap is 5G.
Thanks & Best Regards!
------------------ Original ------------------
From: "Michael McCandless";<lu...@mikemccandless.com>;
Date: Sat, Sep 13, 2014 06:29 PM
To: "Lucene Users"<ja...@lucene.apache.org>;
Subject: Re: OutOfMemoryError throwed by SimpleMergedSegmentWarmer
The warmer just tries to load norms/docValues/etc. for all fields that
have them enabled ... so this is likely telling you an IndexReader
would also hit OOME.
You either need to reduce the number of fields you have indexed, or at
least disable norms (takes 1 byte per doc per indexed field regardless
of whether that doc had indexed that field), or increase HEAP to the
JVM.
Mike McCandless
http://blog.mikemccandless.com
On Sat, Sep 13, 2014 at 4:25 AM, 308181687 <30...@qq.com> wrote:
> Hi, all
> we got an OutOfMemoryError throwed by SimpleMergedSegmentWarmer. We use lucene 4.7， and access index file by NRTCachingDirectory/MMapDirectory. Could any body give me a hand? Strack trace is as follows:
>
>
>
>
>
>
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
>
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:228)
> at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:188)
> at org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:166)
> at org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:519)
> at org.apache.lucene.index.SimpleMergedSegmentWarmer.warm(SimpleMergedSegmentWarmer.java:52)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4275)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3743)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
>
>
>
>
> Thanks & Best Regards!
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: OutOfMemoryError throwed by SimpleMergedSegmentWarmer
Posted by 308181687 <30...@qq.com>.
Hi, Mike
In our use case, we have thousands of index fields, different kind of document have different fields. Do you meant that norms field will consume large memory? Why?
If we decide to disable norms, do we need to rebuild our index entirely? By the way, We have 8 million documents and our jvm heap is 5G.
Thanks & Best Regards!
------------------ Original ------------------
From: "Michael McCandless";<lu...@mikemccandless.com>;
Date: Sat, Sep 13, 2014 06:29 PM
To: "Lucene Users"<ja...@lucene.apache.org>;
Subject: Re: OutOfMemoryError throwed by SimpleMergedSegmentWarmer
The warmer just tries to load norms/docValues/etc. for all fields that
have them enabled ... so this is likely telling you an IndexReader
would also hit OOME.
You either need to reduce the number of fields you have indexed, or at
least disable norms (takes 1 byte per doc per indexed field regardless
of whether that doc had indexed that field), or increase HEAP to the
JVM.
Mike McCandless
http://blog.mikemccandless.com
On Sat, Sep 13, 2014 at 4:25 AM, 308181687 <30...@qq.com> wrote:
> Hi, all
> we got an OutOfMemoryError throwed by SimpleMergedSegmentWarmer. We use lucene 4.7， and access index file by NRTCachingDirectory/MMapDirectory. Could any body give me a hand? Strack trace is as follows:
>
>
>
>
>
>
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
>
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:228)
> at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:188)
> at org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:166)
> at org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:519)
> at org.apache.lucene.index.SimpleMergedSegmentWarmer.warm(SimpleMergedSegmentWarmer.java:52)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4275)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3743)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
>
>
>
>
> Thanks & Best Regards!
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: OutOfMemoryError throwed by SimpleMergedSegmentWarmer
Posted by Michael McCandless <lu...@mikemccandless.com>.
The warmer just tries to load norms/docValues/etc. for all fields that
have them enabled ... so this is likely telling you an IndexReader
would also hit OOME.
You either need to reduce the number of fields you have indexed, or at
least disable norms (takes 1 byte per doc per indexed field regardless
of whether that doc had indexed that field), or increase HEAP to the
JVM.
Mike McCandless
http://blog.mikemccandless.com
On Sat, Sep 13, 2014 at 4:25 AM, 308181687 <30...@qq.com> wrote:
> Hi, all
> we got an OutOfMemoryError throwed by SimpleMergedSegmentWarmer. We use lucene 4.7， and access index file by NRTCachingDirectory/MMapDirectory. Could any body give me a hand? Strack trace is as follows:
>
>
>
>
>
>
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
>
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:228)
> at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:188)
> at org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:166)
> at org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:519)
> at org.apache.lucene.index.SimpleMergedSegmentWarmer.warm(SimpleMergedSegmentWarmer.java:52)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4275)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3743)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
>
>
>
>
> Thanks & Best Regards!
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org