You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "Jason Gerlowski (Jira)" <ji...@apache.org> on 2025/10/09 13:52:00 UTC

[jira] [Commented] (SOLR-6213) StackOverflowException in Solr cloud's leader election

    [ https://issues.apache.org/jira/browse/SOLR-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18028782#comment-18028782 ] 

Jason Gerlowski commented on SOLR-6213:
---------------------------------------

Saw this recently in Solr 9.7 during a period of ZK misconfiguration.  The stacktrace looks a bit different, sharing below:

{code}
o.a.s.c.LeaderElector Failed setting watch => org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /collections/collNameRedacted/leader_elect/shard1/election/7423158284677299411-core_node9-n_0000000118
  at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /collections/collNameRedacted/leader_elect/shard1/election/7423158284677299411-core_node9-n_0000000118
  at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) ~[?:?]
  at org.apache.zookeeper.KeeperException.create(KeeperException.java:53) ~[?:?]
  at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1972) ~[?:?]
  at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$6(SolrZkClient.java:448) ~[?:?]
  at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70) ~[?:?]
  at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:448) ~[?:?]
  at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:164) ~[?:?]
  at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:164) ~[?:?]
  at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:164) ~[?:?]
  .....<300-ish identical lines removed for brevity>
  at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:164) ~[?:?]
  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:311) ~[?:?]
  at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1597) ~[?:?]
  at org.apache.solr.cloud.ZkController.register(ZkController.java:1303) ~[?:?]
  at org.apache.solr.cloud.ZkController.register(ZkController.java:1237) ~[?:?]
  at org.apache.solr.core.ZkContainer.lambda$registerInZk$1(ZkContainer.java:218) ~[?:?]
  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449) ~[?:?]
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
  at java.base/java.lang.Thread.run(Unknown Source) [?:?]
{code}

The root-cause here seems hard to resolve: we probably need to keep retrying replica-registration for cases like transient network-issues, so that Solr can get healthy again on its own after the network issue resolves.  But there's a lot of room to tweak how the retries happen, to avoid a stackoverflow: looping instead of recursion, adding a backoff between retries after the first, say, 100 attempts, etc.

> StackOverflowException in Solr cloud's leader election
> ------------------------------------------------------
>
>                 Key: SOLR-6213
>                 URL: https://issues.apache.org/jira/browse/SOLR-6213
>             Project: Solr
>          Issue Type: Bug
>          Components: SolrCloud
>    Affects Versions: 4.10, 6.0
>            Reporter: Dawid Weiss
>            Priority: Critical
>         Attachments: stackoverflow.txt
>
>
> This is what's causing test hangs (at least on FreeBSD, LUCENE-5786), possibly on other machines too. The problem is stack overflow from looped calls in:
> {code}
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
>   > org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>   > org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
>   > org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>   > org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
>   > org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>   > org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
>   > org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>   > org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
>   > org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>   > org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
>   > org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>   > org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
>   > org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>   > org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
>   > org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>   > org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
>   > org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>   > org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
>   > org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>   > org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:448)
> {code}
> These routines attempt to log information to loggers, which in turn attempts to serialize messages back to the master (test process). When the stack is exhausted the serialization process fails and breaks the communication with the master test node.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org