Memory issue in merge thread

[Thomas Mortagne <th...@gmail.com>]

Hi guys,

I recently upgraded from Solr 4.0 to 4.8.1. I start it with a clean
index (we did some change in the Solr schema in the meantime) and
after some time of indexing a very big database my instance is
becoming totally unusable with 99% of the heap filled. Then when I
restart it it get stuck very quickly with the same memory issue so it
seems linked to the size of the Lucene index more that the time spend
indexing data.

Youtkit is telling me that "Lucene Merge Thread #1"
(ConcurrentMergeScheduler$MergeThread) is keeping 4,095 instances of
org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer$2 in some
local variable(s) which retain about 422MB of RAM and keep adding more
of those to the heap.

Whatever this thread is doing (ConcurrentMergeScheduler javadoc is not
very detailed) it does not seems to be doing it in a very streamed
fashion (if not "simply" a memory leak in 4.8.1). Any idea if this is
expected ? Do I have a way to control the size of the heap this thread
is going to need ?

I have the heap dump if anyone want more details or want to look at
it. The Solr setup can be seen on
https://github.com/xwiki/xwiki-platform/tree/xwiki-platform-6.2/xwiki-platform-core/xwiki-platform-search/xwiki-platform-search-solr/xwiki-platform-search-solr-api/src/main/resources/solr/xwiki/conf

I know all that is only talking about Lucene classes but since on my
side what I use is Solr I tough it was better to ask on this mailing
list.

Thanks,
-- 
Thomas