You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "Yifei Geng (Jira)" <ji...@apache.org> on 2026/06/08 05:49:00 UTC
[jira] [Comment Edited] (SOLR-18273) Solr 10.0.0 treats short ZooKeeper leader failover as session-expiration recovery and removes/re-registers live_nodes
[ https://issues.apache.org/jira/browse/SOLR-18273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18085725#comment-18085725 ]
Yifei Geng edited comment on SOLR-18273 at 6/8/26 5:48 AM:
-----------------------------------------------------------
I looked into the Solr 10.0.0 source code and the observed behavior appears to come from how ZooKeeper connection state events are mapped into ZkController recovery logic.
The relevant code paths are:
* org.apache.solr.common.cloud.OnDisconnect
* org.apache.solr.common.cloud.OnReconnect
* org.apache.solr.cloud.ZkController
In OnDisconnect, both Curator SUSPENDED and LOST events call ZkController.onDisconnect(boolean sessionExpired):
{noformat}
if (newState == ConnectionState.LOST || newState == ConnectionState.SUSPENDED) {
onDisconnect(newState == ConnectionState.LOST);
}
{noformat}
So SUSPENDED is passed as sessionExpired=false, while LOST is passed as sessionExpired=true.
However, OnReconnect does not carry this distinction forward. Any Curator RECONNECTED event calls ZkController.onReconnect():
{noformat}
if (ConnectionState.RECONNECTED.equals(newState)) {
onReconnect();
}
{noformat}
Then ZkController.onReconnect() unconditionally executes logic that is explicitly written as session-expiration recovery:
{noformat}
log.info("ZooKeeper session re-connected ... refreshing core states after session expiration.");
removeEphemeralLiveNode();
zkStateReader.createClusterStateWatchersAndUpdate();
...
registerAllCoresAsDown(false);
createEphemeralLiveNode();
...
executorService.submit(new RegisterCoreAsync(descriptor, true, true));
{noformat}
The third argument to RegisterCoreAsync is afterExpiration=true, so all local cores are re-registered as if the ZooKeeper session had expired.
This explains the observed log sequence:
{noformat}
State change: SUSPENDED
State change: RECONNECTED
ZooKeeper session re-connected ... refreshing core states after session expiration.
Remove node as live in ZooKeeper:/live_nodes/...
Register node as live in ZooKeeper:/live_nodes/...
Registering core ... afterExpiration? true
{noformat}
The important point is that this sequence can happen after a short ZooKeeper leader failover where the client session has not actually expired. In our logs, ZooKeeper leader election completed within a few seconds, while ZooKeeper server-side session expiration messages appeared later. So Solr appears to be entering the session-expiration recovery path based only on RECONNECTED, not based on confirmed session expiration.
There is also an important subtlety: simply returning early from ZkController.onReconnect() when the previous disconnect was only SUSPENDED may not be sufficient, because ZkController.onDisconnect(false) already performs non-trivial local cleanup:
{noformat}
overseer.close();
for (CoreDescriptor descriptor : descriptors) {
closeExistingElectionContext(descriptor, sessionExpired);
}
for (CoreDescriptor descriptor : descriptors) {
descriptor.getCloudDescriptor().setLeader(false);
descriptor.getCloudDescriptor().setHasRegistered(false);
}
{noformat}
For sessionExpired=false, closeExistingElectionContext() closes the local election context but does not remove it from the map, because the ephemeral election znodes should still exist:
{noformat}
prevContext.close();
if (sessionExpired) {
electionContexts.remove(contextKey);
}
{noformat}
So the fix likely needs to treat connection state as an explicit state machine, not as two independent callbacks.
A safer approach would be:
* Track the last known ZooKeeper session id.
* Track whether the last disconnect was LOST/session-expired or only SUSPENDED.
* On RECONNECTED:
* if the previous state was LOST, or if the ZooKeeper session id changed, run the existing full session-expiration recovery path.
* otherwise, run only a lightweight reconnect path:
* refresh/recreate watchers if needed
* refresh cluster state
* restore local state/election handling without removing /live_nodes
* do not call registerAllCoresAsDown(false)
* do not re-register every core with afterExpiration=true
This would preserve the existing full recovery behavior for real session expiration, while avoiding unnecessary live_nodes removal and mass core re-registration during short ZooKeeper leader failovers.
15:43
was (Author: JIRAUSER313539):
I looked into the Solr 10.0.0 source code and the observed behavior appears to come from how ZooKeeper connection state events are mapped into ZkController recovery logic.
The relevant code paths are:
* org.apache.solr.common.cloud.OnDisconnect
* org.apache.solr.common.cloud.OnReconnect
* org.apache.solr.cloud.ZkController
In OnDisconnect, both Curator SUSPENDED and LOST events call ZkController.onDisconnect(boolean sessionExpired):
{noformat}
if (newState == ConnectionState.LOST || newState == ConnectionState.SUSPENDED) {
onDisconnect(newState == ConnectionState.LOST);
}
{noformat}
So SUSPENDED is passed as sessionExpired=false, while LOST is passed as sessionExpired=true.
However, OnReconnect does not carry this distinction forward. Any Curator RECONNECTED event calls ZkController.onReconnect():
{noformat}
if (ConnectionState.RECONNECTED.equals(newState)) {
onReconnect();
}
{noformat}
Then ZkController.onReconnect() unconditionally executes logic that is explicitly written as session-expiration recovery:
{noformat}
log.info("ZooKeeper session re-connected ... refreshing core states after session expiration.");
removeEphemeralLiveNode();
zkStateReader.createClusterStateWatchersAndUpdate();
...
registerAllCoresAsDown(false);
createEphemeralLiveNode();
...
executorService.submit(new RegisterCoreAsync(descriptor, true, true));
{noformat}
The third argument to RegisterCoreAsync is afterExpiration=true, so all local cores are re-registered as if the ZooKeeper session had expired.
This explains the observed log sequence:
{noformat}
State change: SUSPENDED
State change: RECONNECTED
ZooKeeper session re-connected ... refreshing core states after session expiration.
Remove node as live in ZooKeeper:/live_nodes/...
Register node as live in ZooKeeper:/live_nodes/...
Registering core ... afterExpiration? true
{noformat}
The important point is that this sequence can happen after a short ZooKeeper leader failover where the client session has not actually expired. In our logs, ZooKeeper leader election completed within a few seconds, while ZooKeeper server-side session expiration messages appeared later. So Solr appears to be entering the session-expiration recovery path based only on RECONNECTED, not based on confirmed session expiration.
There is also an important subtlety: simply returning early from ZkController.onReconnect() when the previous disconnect was only SUSPENDED may not be sufficient, because ZkController.onDisconnect(false) already performs non-trivial local cleanup:
{noformat}
overseer.close();
for (CoreDescriptor descriptor : descriptors) {
closeExistingElectionContext(descriptor, sessionExpired);
}
for (CoreDescriptor descriptor : descriptors) {
descriptor.getCloudDescriptor().setLeader(false);
descriptor.getCloudDescriptor().setHasRegistered(false);
}
{noformat}
For sessionExpired=false, closeExistingElectionContext() closes the local election context but does not remove it from the map, because the ephemeral election znodes should still exist:
{noformat}
prevContext.close();
if (sessionExpired) {
electionContexts.remove(contextKey);
}
{noformat}
So the fix likely needs to treat connection state as an explicit state machine, not as two independent callbacks.
A safer approach would be:
* Track the last known ZooKeeper session id.
* Track whether the last disconnect was LOST/session-expired or only SUSPENDED.
* On RECONNECTED:
* if the previous state was LOST, or if the ZooKeeper session id changed, run the existing full session-expiration recovery path.
* otherwise, run only a lightweight reconnect path:
* refresh/recreate watchers if needed
* refresh cluster state
* restore local state/election handling without removing /live_nodes
* do not call registerAllCoresAsDown(false)
* do not re-register every core with afterExpiration=true
This would preserve the existing full recovery behavior for real session expiration, while avoiding unnecessary live_nodes removal and mass core re-registration during short ZooKeeper leader failovers.
15:43
默认权限
> Solr 10.0.0 treats short ZooKeeper leader failover as session-expiration recovery and removes/re-registers live_nodes
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18273
> URL: https://issues.apache.org/jira/browse/SOLR-18273
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
> Affects Versions: 10.0
> Environment: Solr 10.0.0
> Reporter: Yifei Geng
> Priority: Critical
>
> ```markdown
> Title: Solr 10.0.0 treats short ZooKeeper leader failover as session-expiration recovery and removes/re-registers live_nodes
> ### Description
> We are observing a SolrCloud stability issue after upgrading/testing with Solr 10.0.0.
> When the ZooKeeper leader node is manually stopped, ZooKeeper elects a new leader successfully within about 2.4 seconds. However, multiple Solr 10 nodes enter a disruptive recovery path after ZooKeeper reconnect: they log `ZooKeeper session re-connected ... refreshing core states after session expiration`, remove their own `/live_nodes` entry, publish themselves as `DOWN`, and then re-register hundreds of cores with `afterExpiration? true`.
> This causes SolrCloud-level instability: `live_nodes` briefly drops from `3 -> 2 -> 1 -> 0`, Overseer leadership changes, shard leader state becomes inconsistent, and update requests start failing with 503 errors.
> The same ZooKeeper leader failover scenario did not cause this behavior in our Solr 8 cluster.
> ### Environment
> - Solr version: `10.0.0`
> - SolrCloud cluster: 3 Solr nodes
> - ZooKeeper version: tested with `3.9.4`
> - ZooKeeper ensemble: 3 nodes
> - ZooKeeper settings:
> - `tickTime=2000`
> - `minSessionTimeout=4000`
> - `maxSessionTimeout=40000`
> - `initLimit=10`
> - `syncLimit=5`
> - Solr JVM / ZK options:
> - `-Dsolr.zookeeper.client.timeout=30000`
> - `-DzkClientTimeout=30000`
> - `-DzkHost=zk-solr-test1:2181,zk-solr-test2:2181,zk-solr-test3:2181/solr10-test`
> ### Steps to reproduce
> 1. Start a 3-node Solr 10.0.0 SolrCloud cluster.
> 2. Start a 3-node ZooKeeper ensemble.
> 3. Run write/update traffic against collections in SolrCloud.
> 4. Manually stop the current ZooKeeper leader process.
> 5. Observe Solr logs and `/live_nodes`.
> ### Expected behavior
> ZooKeeper leader failover should cause a short `SUSPENDED` period and then Solr should reconnect without treating the event as a full session-expiration recovery, as long as the ZooKeeper session has not actually expired.
> Solr nodes should not remove their own live node entries and re-register all cores after a short ZooKeeper leader election.
> ### Actual behavior
> ZooKeeper leader election completed quickly:
> ```text
> 2026-05-29 14:43:25,444 Disconnected from leader zk-solr-test2
> 2026-05-29 14:43:27,969 LEADING - LEADER ELECTION TOOK - 2417 MS
> ```
> But Solr nodes entered a disruptive recovery path.
> Example from Solr node 2:
> ```text
> 2026-05-29 14:43:25.467 WARN Session 0x301ca123e240004 ... EndOfStreamException
> 2026-05-29 14:43:25.581 INFO State change: SUSPENDED
> 2026-05-29 14:43:26.180 WARN zk-solr-test2:2181 ... Connection refused
> 2026-05-29 14:43:28.518 INFO State change: RECONNECTED
> 2026-05-29 14:43:28.519 INFO ZooKeeper session re-connected ... refreshing core states after session expiration.
> 2026-05-29 14:43:28.521 INFO Remove node as live in ZooKeeper:/live_nodes/solr10-test-2:8983_solr
> 2026-05-29 14:43:29.495 INFO Publish node=solr10-test-2:8983_solr as DOWN
> 2026-05-29 14:43:30.042 INFO Register node as live in ZooKeeper: /live_nodes/solr10-test-2:8983_solr
> 2026-05-29 14:43:30.066 INFO Registering core antibody_shard1_replica_n2 afterExpiration? true
> ```
> During this period, live nodes changed as follows:
> ```text
> Updated live nodes from ZooKeeper... (3) -> (2)
> Updated live nodes from ZooKeeper... (2) -> (1)
> Updated live nodes from ZooKeeper... (1) -> (0)
> Updated live nodes from ZooKeeper... (0) -> (1)
> ```
> Update requests failed during and after this state transition:
> ```text
> Cannot talk to ZooKeeper - Updates are disabled.
> ```
> Later, Solr reported leader state inconsistency:
> ```text
> ClusterState says we are the leader, but locally we don't think so.
> ```
> A second Solr node showed the same pattern:
> ```text
> 2026-05-29 14:43:25.616 INFO State change: SUSPENDED
> 2026-05-29 14:43:29.583 INFO State change: RECONNECTED
> 2026-05-29 14:43:29.584 INFO ZooKeeper session re-connected ... refreshing core states after session expiration.
> 2026-05-29 14:43:29.589 INFO Remove node as live in ZooKeeper:/live_nodes/solr10-test-3:8983_solr
> 2026-05-29 14:43:31.040 INFO Publish node=solr10-test-3:8983_solr as DOWN
> ```
> ### Important detail
> ZooKeeper server-side session expiration happened much later than the Solr-side "after session expiration" log.
> ZooKeeper logs:
> ```text
> 2026-05-29 14:47:44,327 Expiring session 0x301ca123e240005, timeout of 30000ms exceeded
> 2026-05-29 14:47:46,327 Expiring session 0x301ca123e240004, timeout of 30000ms exceeded
> ```
> So at `14:43:28/29`, Solr appears to be executing session-expiration recovery even though ZooKeeper had not yet logged actual server-side expiration for those sessions.
> ### Impact
> A short ZooKeeper leader failover is amplified into a SolrCloud-wide instability event:
> - Multiple Solr nodes publish themselves as `DOWN`
> - `/live_nodes` temporarily drops to zero or near-zero
> - Overseer leadership changes
> - Hundreds of cores are re-registered
> - Updates fail with 503
> - Shard leader state becomes inconsistent
> ### Question / suspected issue
> Is Solr 10.0.0 expected to run full session-expiration recovery on `RECONNECTED` after a short ZooKeeper leader failover?
> This looks like Solr may be treating a transient ZooKeeper reconnect as session expiration, or otherwise overreacting during Curator/ZooKeeper reconnect handling, causing unnecessary live node removal and core re-registration.
> Could this be a Solr 10.0.0 regression in ZooKeeper reconnect/session-expiration handling compared with Solr 8?
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org