You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Mike Drob (Jira)" <ji...@apache.org> on 2020/10/26 19:12:00 UTC
[jira] [Comment Edited] (SOLR-14940) ReplicationHandler memory leak
through SolrCore.closeHooks
[ https://issues.apache.org/jira/browse/SOLR-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220941#comment-17220941 ]
Mike Drob edited comment on SOLR-14940 at 10/26/20, 7:11 PM:
-------------------------------------------------------------
It looks like we end up registering ReplicationHandler twice -
On core creation,
{noformat}
at org.apache.solr.handler.ReplicationHandler.registerCloseHook(ReplicationHandler.java:1410) [main/:?]
at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1235) [main/:?]
at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:674) [main/:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1037) [main/:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:929) [main/:?]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1375) [main/:?]
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1261) [main/:?]
{noformat}
And when we register in ZK
{noformat}
at org.apache.solr.handler.ReplicationHandler.registerCloseHook(ReplicationHandler.java:1410) [main/:?]
at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1235) [main/:?]
at org.apache.solr.cloud.ReplicateFromLeader.startReplication(ReplicateFromLeader.java:109) [main/:?]
at org.apache.solr.cloud.ZkController.startReplicationFromLeader(ZkController.java:1305) [main/:?]
at org.apache.solr.cloud.ZkController.register(ZkController.java:1198) [main/:?]
at org.apache.solr.cloud.ZkController.register(ZkController.java:1145) [main/:?]
at org.apache.solr.core.ZkContainer.lambda$registerInZk$1(ZkContainer.java:208) [main/:?]
at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:237) [main/:?]
at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:1195) [main/:?]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1385) [main/:?]
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1261) [main/:?]
{noformat}
I think this is likely a bug.
was (Author: mdrob):
It looks like we end up registering ReplicationHandler twice -
On core creation,
{noformat}
at org.apache.solr.handler.ReplicationHandler.registerCloseHook(ReplicationHandler.java:1410) [main/:?]
at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1235) [main/:?]
at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:674) [main/:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1037) [main/:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:929) [main/:?]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1375) [main/:?]
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1261) [main/:?]
{noformat}
And when we register in ZK
{noformat}
at org.apache.solr.handler.ReplicationHandler.registerCloseHook(ReplicationHandler.java:1410) [main/:?]
at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1235) [main/:?]
at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:674) [main/:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1037) [main/:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:929) [main/:?]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1375) [main/:?]
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1261) [main/:?]
{noformat}
I think this is likely a bug.
> ReplicationHandler memory leak through SolrCore.closeHooks
> ----------------------------------------------------------
>
> Key: SOLR-14940
> URL: https://issues.apache.org/jira/browse/SOLR-14940
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: replication (java)
> Environment: Solr Cloud Cluster on v.8.6.2 configured as 3 TLOG nodes with 2 cores in each JVM.
>
> Reporter: Anver Sotnikov
> Priority: Major
> Attachments: Actual references to hooks that in turn hold references to ReplicationHandlers.png, Memory Analyzer SolrCore.closeHooks .png
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> We are experiencing a memory leak in Solr Cloud cluster configured as 3 TLOG nodes.
> Leader does not seem to be affected while Followers are.
>
> Looking at memory dump we noticed that SolrCore holds lots of references to ReplicationHandler through anonymous inner classes in SolrCore.closeHooks, which in turn holds ReplicationHandlers.
> ReplicationHandler registers hooks as anonymous inner classes in SolrCore.closeHooks through ReplicationHandler.inform() -> ReplicationHandler.registerCloseHook().
>
> Whenever ZkController.stopReplicationFromLeader is called - it would shutdown ReplicationHandler (ReplicationHandler.shutdown()), BUT reference to ReplicationHandler will stay in SolrCore.closeHooks. Once replication is started again on same SolrCore - new ReplicationHandler will be created and registered in closeHooks.
>
> It looks like there are few scenarios when replication is stopped and restarted on same core and in our TLOG setup it shows up quite often.
>
> Potential solutions:
> # Allow unregistering SolrCore.closeHooks so it can be used from ReplicationHandler.shutdown
> # Hack but easier - break the link between ReplicationHandler close hooks and full ReplicationHandler object so ReplicationHandler can be GCed even when hooks are still registered in SolrCore.closeHooks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org