You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@lucene.apache.org by tboeghk <gi...@git.apache.org> on 2017/05/04 11:26:36 UTC

[GitHub] lucene-solr pull request #197: [SOLR-10506] Fixes a memory leak in zk schema...

GitHub user tboeghk opened a pull request:

    https://github.com/apache/lucene-solr/pull/197

    [SOLR-10506] Fixes a memory leak in zk schema watching

    Upon manual Solr Collection reloading, references to the closed SolrCore are not fully removed by the garbage collector as a strong reference to the ZkIndexSchemaReader is held in a ZooKeeper Watcher that watches for schema changes.
    
    In our case, this leads to a massive memory leak as managed resources are still referenced by the closed SolrCore. Our Solr cloud environment utilizes rather large managed resources (synonyms, stopwords). To reproduce, we fired out environment up and reloaded the collection 13 times. As a result we fully exhausted our heap. A closer look with the Yourkit profiler revealed 13 SolrCore instances, still holding strong references to the garbage collection root.
    
    Each SolrCore instance holds a single path with strong references to the gc root via a Watcher in ZkIndexSchemaReader. The ZkIndexSchemaReader registers a close hook in the SolrCore but the Zookeeper is not removed upon core close.
    
    We supplied this Github Pull Request that extracts the zookeeper Watcher as a static inner class. To eliminate the memory leak, the schema reader is held inside a WeakReference and the reference is explicitly removed on core close.
    
    Initially I wanted to supply a test case but unfortunately did not find a good starting point ...
    
    N.B.: I did this second PR for the same issue to separate code changes for both SOLR-10506 and SOLR-10550 which I maintained on the same fork branch :-/

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/shopping24/lucene-solr branch_6_5__SOLR-10506

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/lucene-solr/pull/197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #197
    
----
commit aa3a96cd88d1724f3ab641f25c5373ff58f4616d
Author: Torsten Bøgh Köster <to...@s24.com>
Date:   2017-04-18T09:43:52Z

    Fixes a memory leak in zk schema watching

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

[GitHub] lucene-solr issue #197: [SOLR-10506] Fixes a memory leak in zk schema watchi...

Posted by uschindler <gi...@git.apache.org>.

Github user uschindler commented on the issue:

    https://github.com/apache/lucene-solr/pull/197
  
    Should we not have a non-empty message when logging the InterruptedException?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

[GitHub] lucene-solr issue #197: [SOLR-10506] Fixes a memory leak in zk schema watchi...

Posted by tboeghk <gi...@git.apache.org>.

Github user tboeghk commented on the issue:

    https://github.com/apache/lucene-solr/pull/197
  
    +1 
    
    ... but I wanted to focus the PR above on the fix (the empty message is copied code). Also this PR is outdated as the current patch is atached here: https://issues.apache.org/jira/browse/SOLR-10506


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org