You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by ni...@emc.com on 2012/06/02 01:53:48 UTC
OOM during IndexReader open
Hi,
I am getting the following OOM consistently whenever the index is opened . Is it because now the index is holding too many terms ? Our application ( that has Lucene 2.9.3 ) already has reached 1.8G of virtual memory.
Is there a way I can avoid OOM during IndexReader open because of too many terms ?
2012-05-30 14:12:46,139 [MC:10.2.2.1-1332015615933-1264 FS:parinfprdcl01k_655394_0] ERROR util - Context: MC: 10.2.2 .1-1332015615933-1264 failed at the state LocalIndexOptimizeAndCompress, NumActiveThreads = 64. Error: java.lang.OutO fMemoryError: Java heap space % STACK:
org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:90)
org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:122)
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:652)
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:613)
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:106)
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:71)
org.apache.lucene.index.IndexReader.open(IndexReader.java:482)
org.apache.lucene.index.IndexReader.open(IndexReader.java:320)
com.kazeon.search.indexingengine.context.MergerContext.deleteAgainstFiler(MergerContext.java:1357)
com.kazeon.search.indexingengine.context.MergerContext.deleteAgainstFiler(MergerContext.java:656)
com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.modifyState Aux(LocalIndexOptimizeAndCompressModifier.java:303)
com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.mergeAllICs (LocalIndexOptimizeAndCompressModifier.java:175)
com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.modifyState (LocalIndexOptimizeAndCompressModifier.java:106)
com.kazeon.util.scoreboard.WorkerThread.run(WorkerThread.java:31)
thanks,
Nishesh
Re: RAMDirectory with FSDirectory merging Versus large mergeFactor
and RAMBufferSizeMB
Posted by Lance Norskog <go...@gmail.com>.
Ramdirectory is no longer an interesting technique for this. It makes
garbage collection do a lot of work. With memory-mapped directory the
data is cached by the OS instead of Java, and OS is very good at this.
TieredMergePolicy is much smarter about time spent merging segments.
Lucene In Action 2 might be more help than a 6-year-old book :)
On Mon, Jun 4, 2012 at 12:47 AM, Maxim Terletsky <sx...@yahoo.com> wrote:
> Hi guys,
> There are two approaches I see in Lucene In Action about speeding up the indexing process.
>
> 1) Simply increase the mergeFactor and RAMBufferSizeMB.
> 2) Using RAMDirectory as a buffer (perhaps even several in parallel) and later merging it using addIndexes to FSDirectory.
>
> So my question is the following:
> In case I have only 1 thread with RAMDirectory - is that pretty much the same as method 1? Since it's in memory anyhow for large mergeFactor and large RAMBufferSizeMB.
>
> Maxim
>
>
> ________________________________
--
Lance Norskog
goksron@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RAMDirectory with FSDirectory merging Versus large mergeFactor and RAMBufferSizeMB
Posted by Maxim Terletsky <sx...@yahoo.com>.
Hi guys,
There are two approaches I see in Lucene In Action about speeding up the indexing process.
1) Simply increase the mergeFactor and RAMBufferSizeMB.
2) Using RAMDirectory as a buffer (perhaps even several in parallel) and later merging it using addIndexes to FSDirectory.
So my question is the following:
In case I have only 1 thread with RAMDirectory - is that pretty much the same as method 1? Since it's in memory anyhow for large mergeFactor and large RAMBufferSizeMB.
Maxim
________________________________
Re: OOM during IndexReader open
Posted by Michael McCandless <lu...@mikemccandless.com>.
It could be your index has an unusual number of unique terms.
If you can upgrade to the latest 3.x, the RAM used by the terms index
has been very substantially reduced...
If not, try setting the termInfosIndexDivisor to eg 2 or 3 ... this
will load 1/2 or 1/3 of the indexed terms into RAM, but may slow down
certain searches...
Mike McCandless
http://blog.mikemccandless.com
On Sat, Jun 2, 2012 at 12:02 PM, Jack Krupansky <ja...@basetechnology.com> wrote:
> Have you tried a System.gc before opening the index? And check how much
> memory is available before the open. Does the open seem to be taking more
> memory that could reasonably be expected? How big is the index, how many
> docs, how many unique terms in that index?
>
> 1.8G doesn't sound like a lot. You may simply have to bite the bullet and
> move to an environment with more memory. Or, split the index and run on
> multiple machines/JVMs.
>
> Have many deletes been performed? How many segments are there? Maybe you can
> merge using a standalone program that doesn't have the rest of your app
> consuming memory.
>
> -- Jack Krupansky
>
> -----Original Message----- From: nishesh.gupta@emc.com
> Sent: Friday, June 01, 2012 7:53 PM
> To: java-user@lucene.apache.org
> Subject: OOM during IndexReader open
>
>
> Hi,
>
> I am getting the following OOM consistently whenever the index is opened .
> Is it because now the index is holding too many terms ? Our application (
> that has Lucene 2.9.3 ) already has reached 1.8G of virtual memory.
> Is there a way I can avoid OOM during IndexReader open because of too many
> terms ?
>
>
> 2012-05-30 14:12:46,139 [MC:10.2.2.1-1332015615933-1264
> FS:parinfprdcl01k_655394_0] ERROR util - Context: MC: 10.2.2
> .1-1332015615933-1264 failed at the state LocalIndexOptimizeAndCompress,
> NumActiveThreads = 64. Error: java.lang.OutO fMemoryError: Java heap
> space % STACK:
> org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:90)
> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:122)
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:652)
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:613)
> org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:106)
> org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:71)
> org.apache.lucene.index.IndexReader.open(IndexReader.java:482)
> org.apache.lucene.index.IndexReader.open(IndexReader.java:320)
> com.kazeon.search.indexingengine.context.MergerContext.deleteAgainstFiler(MergerContext.java:1357)
> com.kazeon.search.indexingengine.context.MergerContext.deleteAgainstFiler(MergerContext.java:656)
> com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.modifyState
> Aux(LocalIndexOptimizeAndCompressModifier.java:303)
> com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.mergeAllICs
> (LocalIndexOptimizeAndCompressModifier.java:175)
> com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.modifyState
> (LocalIndexOptimizeAndCompressModifier.java:106)
> com.kazeon.util.scoreboard.WorkerThread.run(WorkerThread.java:31)
>
>
> thanks,
> Nishesh
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: OOM during IndexReader open
Posted by Jack Krupansky <ja...@basetechnology.com>.
Have you tried a System.gc before opening the index? And check how much
memory is available before the open. Does the open seem to be taking more
memory that could reasonably be expected? How big is the index, how many
docs, how many unique terms in that index?
1.8G doesn't sound like a lot. You may simply have to bite the bullet and
move to an environment with more memory. Or, split the index and run on
multiple machines/JVMs.
Have many deletes been performed? How many segments are there? Maybe you can
merge using a standalone program that doesn't have the rest of your app
consuming memory.
-- Jack Krupansky
-----Original Message-----
From: nishesh.gupta@emc.com
Sent: Friday, June 01, 2012 7:53 PM
To: java-user@lucene.apache.org
Subject: OOM during IndexReader open
Hi,
I am getting the following OOM consistently whenever the index is opened .
Is it because now the index is holding too many terms ? Our application (
that has Lucene 2.9.3 ) already has reached 1.8G of virtual memory.
Is there a way I can avoid OOM during IndexReader open because of too many
terms ?
2012-05-30 14:12:46,139 [MC:10.2.2.1-1332015615933-1264
FS:parinfprdcl01k_655394_0] ERROR util - Context: MC: 10.2.2
.1-1332015615933-1264 failed at the state LocalIndexOptimizeAndCompress,
NumActiveThreads = 64. Error: java.lang.OutO fMemoryError: Java heap
space % STACK:
org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:90)
org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:122)
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:652)
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:613)
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:106)
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:71)
org.apache.lucene.index.IndexReader.open(IndexReader.java:482)
org.apache.lucene.index.IndexReader.open(IndexReader.java:320)
com.kazeon.search.indexingengine.context.MergerContext.deleteAgainstFiler(MergerContext.java:1357)
com.kazeon.search.indexingengine.context.MergerContext.deleteAgainstFiler(MergerContext.java:656)
com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.modifyState
Aux(LocalIndexOptimizeAndCompressModifier.java:303)
com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.mergeAllICs
(LocalIndexOptimizeAndCompressModifier.java:175)
com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.modifyState
(LocalIndexOptimizeAndCompressModifier.java:106)
com.kazeon.util.scoreboard.WorkerThread.run(WorkerThread.java:31)
thanks,
Nishesh
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org