[lucene-6.3.0] hit tragic OutOfMemoryError inside getReader

["sandesh.yapuram" <sa...@seclore.com>]

I'm trying to index 5M Documents and currently indexing takes 13 hours. My
data source is SQL db and I've verified out of 13 hours only 30 mins is
spent in fetching data so SQL is not the bottleneck. The complete index size
is 40GB.

My application runs on Tomcat with 2GB JVM space.

Just after the indexing process is complete, when I try to run a simple
query, the application hits OutOfMemoryError: Java Heap Space

The InfoReader log reports 'hit exception during NRT Reader'
<http://lucene.472066.n3.nabble.com/file/n4345589/exception_during_nrt_reader.png> 

Also around the same time frame the InfoReader reports following for
updateMergeThreads
<http://lucene.472066.n3.nabble.com/file/n4345589/updateMergeThreads.png> 

I analyzed the logs and found that the merging process is painfully slow.
I'm using the Default IndexWriterConfig settings.

I've tried indexing same 5m Docs on 3GB JVM with a SerialMergeScheduler with
setRAMBufferSizeMB=48 and setMaxMergedSizeMB=2G in TieredMergePolicy which
takes 14Hours.

Increasing the tomcat jvm space may not always be an option, so is there any
way to tune the Merging process to make it faster and also not run out of
memory.








--
View this message in context: http://lucene.472066.n3.nabble.com/lucene-6-3-0-hit-tragic-OutOfMemoryError-inside-getReader-tp4345589.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org