[GitHub] lucene-solr pull request #198: [SOLR-10550] Improve FileFloatSource eviction...

[tboeghk <gi...@git.apache.org>]

GitHub user tboeghk opened a pull request:

    https://github.com/apache/lucene-solr/pull/198

    [SOLR-10550] Improve FileFloatSource eviction // reduce FileFloatSource memory footprint

    As a follow up from `SOLR-10506` we found another possible memory leak in Solr. The values generated from an `ExternalFileField` are cached in a static cache inside the `FileFloatSource`. That cache caches both a `IndexReader` and `FileFloatSource`s loaded using that `IndexReader`.
    Cache eviction is left to the internally used WeakHashMap or a full eviction can be triggered via url. We are dealing with large synonym files and word lists stored in managed resources. Those are tied to the SolrCore as described in `SOLR-10506`. We're also using `ExternalFileField`s whose `FileFloatSource` are cached in said static cache. The FileFloatSource hold strong (transitive) references to the SolrCore they have been created for.
    
    After a couple of collection reloads, the cache eviction mechanism of the `WeakHashMap` gets activated pretty close to heap exhaustion. The patch attached adds a mechanism to evict cache entries created in the context of a `SolrCore` upon it's close using a close hook in the `ExternalFileFieldReloader`. It furthermore adds a static cache reset method for all entries bound to a given IndexReader. I'm not sure, if the added cache resets are too aggressive or executed too often, I'd like to leave that to the experts.
    
    N.B.: I did this second PR for the same issue to separate code changes for both SOLR-10506 and SOLR-10550 which I maintained on the same fork branch :-/


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/shopping24/lucene-solr branch_6_5__SOLR-10550

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/lucene-solr/pull/198.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #198
    
----
commit 024e950f840b8466272014e4594e198560760c79
Author: Torsten Bøgh Köster <to...@s24.com>
Date:   2017-04-21T12:51:21Z

    clear cached field sources on core close

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org