You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "Justin Pealing (Jira)" <ji...@apache.org> on 2026/07/23 09:59:00 UTC

[jira] [Updated] (SOLR-18316) Slow Solrcloud recovery following restart of Zookeeper leader

     [ https://issues.apache.org/jira/browse/SOLR-18316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Justin Pealing updated SOLR-18316:
----------------------------------
    Description: 
Since updating to Solrcloud 10 we have been experiencing periods of downtime when the Zookeeper leader restarts. I have a docker-compose file which reproduces this issue, and I believe this is a regression from 9.10.1.
h2. Steps to reproduce

The attached docker-compose.yml creates a standard 2 node Solrcloud cluster with a 3 node Zookeeper, as well as a short script to create some empty collections. (This doesn't exactly match our production environment but I wanted to make sure none of our configuration was causing this issue).

{{docker compose up -d}}
{{Wait for Solr to come up}}
{{docker compose run --rm collections 10}}

Once the collections are up, identify and restart the Zookeeper leader (e.g. docker compose restart zoo1). This results in 1 or more of the Solr nodes showing as "(DEAD)" in the dashboard, and a number of replicas showing as "Down" for a period of time (which may or may not result in collections being unavailable, depending on which nodes are affected), after which the Solrcloud cluster self-recovers.

The amount of downtime appears to be proportional to the number of collections. With 1 collection recovery takes a few seconds, with 10 recovery takes a few minutes. In live we have hundreds of collections and we have seen Solr remain down for hours in this scenario (presumably it would eventually recover). Restarting the down Solr nodes also works for recovery.

This is reproducible with 100% reliability when restarting the Zookeeper leader with the attached docker-compose.yml, but when testing with `solr:9.10.1` (replace the image tags in the docker-compose.yml) there is no perceptible downtime.
h2. Logs

I've not included too many logs here as the issue is reproducible in docker-compose, but I see a large volume of these logs on the affected node:

{{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
{{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
{{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
{{{\{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.839 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
{{{\{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.840 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
{{{\{2026-07-23 10:49:44.841 | 2026-07-23 09:49:44.841 INFO (zkCallback-14-thread-5) [] o.a.s.c.OverseerElectionContext I am going to be the leader 172.21.0.6:8983_solr }}}}
{{{\{2026-07-23 10:49:44.843 | 2026-07-23 09:49:44.843 ERROR (zkCallback-14-thread-5) [] o.a.s.c.LeaderElector node exists =>org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /overseer_elect/leader }}}}
{{{{2026-07-23 10:49:44.843 | at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)}}}}

  was:
Since updating to Solrcloud 10 we have been experiencing periods of downtime when the Zookeeper leader restarts. I have a docker-compose file which reproduces this issue, and I believe this is a regression from 9.10.1.
h2. Steps to reproduce

The attached docker-compose.yml creates a standard 2 node Solrcloud cluster with a 3 node Zookeeper, as well as a short script to create some empty collections. (This doesn't exactly match our production environment but I wanted to make sure none of our configuration was causing this issue).

{{docker compose up -d}}
{{Wait for Solr to come up}}
{{docker compose run --rm collections 10}}

Once the collections are up, identify and restart the Zookeeper leader (e.g. `docker compose restart zoo1`). This results in 1 or more of the Solr nodes showing as "(DEAD)" in the dashboard, and a number of replicas showing as "Down" for a period of time (which may or may not result in collections being unavailable, depending on which nodes are affected), after which the Solrcloud cluster self-recovers.

The amount of downtime appears to be proportional to the number of collections. With 1 collection recovery takes a few seconds, with 10 recovery takes a few minutes. In live we have hundreds of collections and we have seen Solr remain down for hours in this scenario (presumably it would eventually recover). Restarting the down Solr nodes also works for recovery.

This is reproducible with 100% reliability when restarting the Zookeeper leader with the attached docker-compose.yml, but when testing with `solr:9.10.1` (replace the image tags in the docker-compose.yml) there is no perceptible downtime.
h2. Logs

I've not included too many logs here as the issue is reproducible in docker-compose, but I see a large volume of these logs on the affected node:

{{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}
{{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}
{{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}
{{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.839 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}
{{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.840 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}
{{2026-07-23 10:49:44.841 | 2026-07-23 09:49:44.841 INFO (zkCallback-14-thread-5) [] o.a.s.c.OverseerElectionContext I am going to be the leader 172.21.0.6:8983_solr }}
{{2026-07-23 10:49:44.843 | 2026-07-23 09:49:44.843 ERROR (zkCallback-14-thread-5) [] o.a.s.c.LeaderElector node exists =>org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /overseer_elect/leader }}
{{2026-07-23 10:49:44.843 | at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)}}


> Slow Solrcloud recovery following restart of Zookeeper leader
> -------------------------------------------------------------
>
>                 Key: SOLR-18316
>                 URL: https://issues.apache.org/jira/browse/SOLR-18316
>             Project: Solr
>          Issue Type: Bug
>          Components: SolrCloud
>    Affects Versions: 10.0
>         Environment: Docker Desktop (Windows)
>            Reporter: Justin Pealing
>            Priority: Major
>         Attachments: docker-compose.yml
>
>
> Since updating to Solrcloud 10 we have been experiencing periods of downtime when the Zookeeper leader restarts. I have a docker-compose file which reproduces this issue, and I believe this is a regression from 9.10.1.
> h2. Steps to reproduce
> The attached docker-compose.yml creates a standard 2 node Solrcloud cluster with a 3 node Zookeeper, as well as a short script to create some empty collections. (This doesn't exactly match our production environment but I wanted to make sure none of our configuration was causing this issue).
> {{docker compose up -d}}
> {{Wait for Solr to come up}}
> {{docker compose run --rm collections 10}}
> Once the collections are up, identify and restart the Zookeeper leader (e.g. docker compose restart zoo1). This results in 1 or more of the Solr nodes showing as "(DEAD)" in the dashboard, and a number of replicas showing as "Down" for a period of time (which may or may not result in collections being unavailable, depending on which nodes are affected), after which the Solrcloud cluster self-recovers.
> The amount of downtime appears to be proportional to the number of collections. With 1 collection recovery takes a few seconds, with 10 recovery takes a few minutes. In live we have hundreds of collections and we have seen Solr remain down for hours in this scenario (presumably it would eventually recover). Restarting the down Solr nodes also works for recovery.
> This is reproducible with 100% reliability when restarting the Zookeeper leader with the attached docker-compose.yml, but when testing with `solr:9.10.1` (replace the image tags in the docker-compose.yml) there is no perceptible downtime.
> h2. Logs
> I've not included too many logs here as the issue is reproducible in docker-compose, but I see a large volume of these logs on the affected node:
> {{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
> {{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
> {{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
> {{{\{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.839 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
> {{{\{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.840 INFO (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
> {{{\{2026-07-23 10:49:44.841 | 2026-07-23 09:49:44.841 INFO (zkCallback-14-thread-5) [] o.a.s.c.OverseerElectionContext I am going to be the leader 172.21.0.6:8983_solr }}}}
> {{{\{2026-07-23 10:49:44.843 | 2026-07-23 09:49:44.843 ERROR (zkCallback-14-thread-5) [] o.a.s.c.LeaderElector node exists =>org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /overseer_elect/leader }}}}
> {{{{2026-07-23 10:49:44.843 | at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)}}}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org