[jira] [Created] (SOLR-10015) Remove strong reference to Field Cache key (optional) so that GC can release some Field Cache entries when Solr is under memory pressure

["Michael Sun (JIRA)" <ji...@apache.org>]

Michael Sun created SOLR-10015:
----------------------------------

             Summary: Remove strong reference to Field Cache key (optional) so that GC can release some Field Cache entries when Solr is under memory pressure
                 Key: SOLR-10015
                 URL: https://issues.apache.org/jira/browse/SOLR-10015
             Project: Solr
          Issue Type: Improvement
      Security Level: Public (Default Security Level. Issues are Public)
            Reporter: Michael Sun


In current Field Cache (FC) implementation, a WeakHashMap is used, supposedly designed to allow GC to release some Field Cache entries when Solr is under memory pressure. 

However, in practice, FC entry releasing seldom happens. Even worse, sometime Solr goes OOM and heap dump shows large amount of memory is actually used by FC. It's a sign that GC is not able to release FC entries even WeakHashMap is used.

The reason is that FC is using SegmentCoreReaders as the key to the WeakHashMap. However, SegmentCoreReaders is usually strong referenced by SegmentReader. A strong reference would prevent GC to release the key and therefore the value. Therefore GC can't release entries in FC's WeakHashMap. 

The JIRA is to propose a solution to remove the strong reference mentioned above so that GC can release FC entries to avoid long GC pause or OOM. It needs to be optional because this change is a tradeoff, trading more CPU cycles for low memory footage. User can make final decision depending on their use cases.

The prototype attached use a combination of directory name and segment name as key to the WeakHashMap, replacing the SegmentCoreReaders. Without change, Solr doesn't release any FC entries after a GC is manually triggered. With the change, FC entries are usually released after GC.

However, I am not sure if it's the best way to solve this problem. Any suggestions are welcome.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org