Lucene 6.5.1 memory consumption on 64 bit linux System

["ankur.168" <an...@gmail.com>]

Hi, 

I am facing GC issues while fetching search response from Lucene. Below are 
the details of how I am using Lucene. 

Server Details - Linux 64 bit, 12 cores 32 GB Ram 
Total number of applications running on box - 4(there are 2 other 
application running on same box with lesser configurations) 
*JVM 1(to index data)* Details - total - 8 GB, eden=5 GB, S0+S1 = 1 GB and 
Old gen=2 GB 
*JVM 2 (to retrieve data)* Details - total - 8 GB, eden=5 GB, S0+S1 = 1 GB 
and Old gen=2 GB 

I am using *Lucene 6.5.1 to index* my data from DB and then retrieve the 
same. I have 2 different indexes(lets say index 1 and index 2). *Index 1 
size is 1.49 GB* with docs count 436299 and terms count 5517926. *Index 2 
size is 525 MB* with docs count 615357 and terms count 2909071. 
Both indexes are stored using *MMapDirectory* implementation with all fields 
as indexed true. I reindex both indexes twice a day using cron jobs. 

While retrieving I initialize indexReader and IndexSearcher on server 
startup, here again I am using *FSDirectory.open* which in turn create 
MMapdirectory object since I am running this on a 64 bit linux box. There is 
a job which runs every 5 mins to check if new index is created or not, in 
case new index is created I will recreate indexReader and IndexSearcher and 
close the old ones. 

When I search for docs, I am using *SearcherManager.acquire() to obtain a 
new Searcher* Instance and in finally block I am release the same using 
SearcherManager.release(). So basically Searcher is every request based. 

I have noticed that one server startup for *first 2-3 days there are too 
many minor GC* happens but rare Major GC, but *after 2-3 days Major GC 
starts* and sometimes it pauses my application for a minute too. 
*The Eden space is always filled even if I am not firing any requests to the 
application.*
When I took a look at heap dump, its either ScoreDoc or SearcherManager 
instances which is consuming most of the memory. 

Can you guys help me here to understand below things- 
1. why this behaviour is happening? 
2. How lucene manages memory in JVM? 
3. How indexReader and Searcher manager instances consume the memory? 



--
Sent from: http://lucene.472066.n3.nabble.com/Lucene-Java-Developer-f564358.html

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org