You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "gerlowskija (via GitHub)" <gi...@apache.org> on 2024/04/05 21:57:48 UTC
[PR] Support running embedded-zk in "ensemble" mode [solr]
gerlowskija opened a new pull request, #2391:
URL: https://github.com/apache/solr/pull/2391
# Description
Prior to this commit, Solr only supported running embedded-ZK in "standalone" mode (which cannot take part in any larger ZK ensemble or quorum). But there are usecases that would benefit from being able to do this, both on the development and testing side, and event for adventurous users who might want the benefits of a small multi-node SolrCloud cluster without the headache of also deploying ZK.
# Solution
This commit augments our embedded-ZK code to support running embedded-ZK
in "quorum" or ensemble mode. Multiple Solr nodes can now all have
their embedded-ZK's join a multi-node quorum upon startup. Other than
Solr and ZK sharing a process, the embedded- ZK ensemble behaves
identically to one formed of independent processes: nodes can join or
leave the cluster, etc.
Embedded-ensemble-ZK is enabled any time the `zkQuorumRun` system
property is present, along with an explicitly specified ZK host string.
On startup, Solr will identify which host in the zk-conn-string it
should be (based on admittedly hacky heuristics), and then spins up a
'ZooKeeperServerEmbedded' instance in-process to join the ensemble. e.g.
```
export LH="localhost"
# Run each below in a separate terminal, one for each host in the zk-conn string
bin/solr start -p 8983 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun
bin/solr start -p 8984 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun
bin/solr start -p 8985 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun
```
Some notes:
- this doesn't (yet) work with ZK's dynamic-ensemble feature, so all
ZK nodes must be specified in a static ZK conn string provided at
startup
- this appears to run best when the security-manager is disabled.
- the interface in particular for how we expose this is pretty rough, and there's a lot of room for improvement.
# Tests
None, yet.
# Checklist
Please review the following and check all that apply:
- [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [ ] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2737926688
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
Much simpler to split in two separate statefulsets, and let users configure resources individually, and delegate to k8s to keep them alive.
I’m keen to get to a solid MVP in this branch, release it and iterate for each minor release based on feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2737937161
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
Makes sense. Once this gets released, then I'll be ready to merge #2783 and update it to reflect the best practice of "seperate statefulsets"!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2435572085
##########
solr/test-framework/src/java/org/apache/solr/cloud/SolrCloudWithEmbeddedZkQuorumTestCase.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.lang.invoke.MethodHandles;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base class for SolrCloud tests that use embedded ZooKeeper running in quorum mode
Review Comment:
True.
Regarding this base class, we might not need it, as one can extend `SolrCloudTestCase` with a configure statement like this
```java
cluster =
configureCluster(NUM_NODES)
.addConfig("conf1", configPath)
.withEmbeddedZkQuorum()
.build();
```
So I vote for removing it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2435742752
##########
solr/test-framework/src/java/org/apache/solr/cloud/SolrCloudWithEmbeddedZkQuorumTestCase.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.lang.invoke.MethodHandles;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base class for SolrCloud tests that use embedded ZooKeeper running in quorum mode
Review Comment:
agreed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2434251026
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -326,14 +331,17 @@ public SolrMetricsContext getSolrMetricsContext() {
}
}
- private static void startZKSE(int port, String zkHomeDir) throws Exception {
+ private void startZKSE(int port, String zkHomeDir) throws Exception {
Properties p = new Properties();
- p.load(new FileInputStream(zkHomeDir + "/zoo.cfg"));
+ try (FileInputStream fis = new FileInputStream(zkHomeDir + "/zoo.cfg")) {
+ p.load(fis);
+ }
p.setProperty("clientPort", String.valueOf(port));
- // TODO NOCOMMIT - hang onto the created ZooKeeperServerEmbedded to be able to close it
- // gracefully the way we do today with zkServer
- ZooKeeperServerEmbedded.builder().baseDir(Path.of(zkHomeDir)).configuration(p).build().start();
+ zkServerEmbedded =
Review Comment:
Ah ha! The magic!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2736213798
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -192,9 +193,13 @@ public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
// we are ZooKeeper enabled
try {
// If this is an ensemble, allow for a long connect time for other servers to come up
- if (zkServerEnabled && zkServer.getServers().size() > 1) {
+ if (zkServerEnabled && zkServer != null && zkServer.getServers().size() > 1) {
zkClientConnectTimeout = 24 * 60 * 60 * 1000; // 1 day for embedded ensemble
log.info("Zookeeper client={} Waiting for a quorum.", zookeeperHost);
+ } else if (zkServerEnabled && runAsQuorum) {
+ // Quorum mode also needs long timeout for other nodes to start
+ zkClientConnectTimeout = 24 * 60 * 60 * 1000; // 1 day for embedded quorum
Review Comment:
This was an existing timeout variable from 2013 so to keep the changes minimal and easy to review I stuck with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3234373446
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -192,13 +196,62 @@ public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
e);
}
} else if (zkServerEnabled) {
- // Old school ZooKeeperServerMain being used under the covers (bin/solr -c without ZK_HOST).
- zkServer =
- SolrZkServer.createAndStart(config.getZkHost(), solrHome, config.getSolrHostPort());
+ try {
+ final Path zkDataHome =
+ Path.of(
+ EnvUtils.getProperty(
+ "solr.zookeeper.server.datadir", solrHome.resolve("zoo_data").toString()));
Review Comment:
Do we normally keep constants like this inline or do we have somewhere we try to keep all sysprop-names for e.g. zookeeper together in one place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217647289
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -70,42 +81,119 @@ public class ZkContainer {
public ZkContainer() {}
public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
- boolean zkRun = EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ // zkServerEnabled is set whenever in solrCloud mode ('-c') but no explicit zkHost/ZK_HOST is
+ // provided.
+ final boolean zkServerEnabled =
+ EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ boolean zkQuorumNode = false;
+ if (NodeRoles.MODE_ON.equals(cc.nodeRoles.getRoleMode(NodeRoles.Role.ZOOKEEPER_QUORUM))) {
+ zkQuorumNode = true;
+ log.info("Starting node in ZooKeeper Quorum role.");
+ }
- if (zkRun && config == null)
+ if (zkServerEnabled && config == null) {
throw new SolrException(
SolrException.ErrorCode.SERVER_ERROR,
"Cannot start Solr in cloud mode - no cloud config provided");
+ }
- if (config == null) return; // not in zk mode
+ if (config == null) {
+ log.info("Solr is running in standalone mode");
+ return;
+ }
+
+ final boolean runAsQuorum = config.getZkHost() != null && zkQuorumNode;
String zookeeperHost = config.getZkHost();
+ final var solrHome = cc.getSolrHome();
+ if (zkServerEnabled) {
+ if (!runAsQuorum) {
Review Comment:
Fixed. AI assistant refactored relevant code fragments, see the change in this commit https://github.com/apache/solr/pull/2391/changes/47251b9c7cc7904975bb1be303552cbf18fd5fdc?diff=split&w=1 - it looks solid..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217566162
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -313,8 +413,37 @@ public void close() {
zkController.close();
}
} finally {
- if (zkServer != null) {
- zkServer.stop();
+ try {
+ if (zkServer != null) {
+ zkServer.stop();
+ }
+ } finally {
+ if (zkServerEmbedded != null) {
+ try {
+ zkServerEmbedded.close();
+ log.info("Closed embedded ZooKeeper server in quorum mode");
Review Comment:
Close log should come after the wait loop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2486414044
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
I wonder if `-z` should be reserved for connecting to an external ZK quorum, and we invent a new option for telling SolrCloud which solr nodes will run the zk node role? That would make it less confusing, e.g.
```bash
bin/solr start --quorum-nodes=solr-node-1:8983,solr-node-2:8983,solr-node-3:8983
```
That would mean we want Solr to manage our ZK, and translate into the correct `solr.zk.host` setting for each host, no matter if the host you are starting is one of the quorum nodes or not. And if the node you are starting is a quorum node, it will detect that it's own host name matches a quorum host, and add `-Dsolr.node.roles...` during startup, and the ZK bootstrap code will configure its internal ZK config using the `--quorum` / `-Dsolr.quorum.nodes` values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3480335165
I asked Claude to summarize the last seven commits and he said:
_Recent work on the `spike-zk-quorum` branch has focused on refining Solr's embedded ZooKeeper quorum mode implementation through targeted code improvements. The codebase has been streamlined by removing unnecessary variables and conditional logic in the `SolrZkServer` class, particularly eliminating redundant checks around the `zkServerEnabled` flag and renaming methods like `startZKSE` to more descriptive alternatives. Initialization has been optimized by removing the `/solr/initialized` ZooKeeper node, which was only needed for multi-threading protection that's unnecessary in the current implementation. The ZooKeeper initialization logic in `ZkContainer` has been restructured with clearer nesting and better separation between standalone and quorum modes, making the codebase more maintainable. These changes collectively enhance the embedded ZooKeeper quorum functionality while simplifying the code structure, resulting in a more robust and easier-to-understand implementation that
properly manages ZooKeeper server resources throughout Solr's lifecycle._
I think my summary is: **I discovered that we have TWO ways of starting ZK, not including our independent `ZkTestServer` that runs a `ZkTestServer.ZKServerMain` during tests.**
We now have:
* For testing only, we have `org.apache.solr.cloud.ZkTestServer.ZKServerMain` that wraps `org.apache.zookeeper.server.ZooKeeperServer`.
* `org.apache.solr.cloud.SolrZkServer` that wraps either a `org.apache.zookeeper.server.ZooKeeperServer` OR if there is more then one, then does a `org.apache.zookeeper.server.quorum.QuorumPeerMain`
* `org.apache.solr.core.ZkContainer` that starts up `org.apache.zookeeper.server.embedded.ZooKeeperServerEmbedded`
I am now wondering could we convert SolrZkServer to using a `ZooKeeperServerEmbedded` instead of a `ZooKeeperServer`?
Secondly, now seeing the use of `QuorumPeerMain`, which currently kicks in if there are multiple ZooKeeper servers defined (presumably only in an external cluster set up), I wonder if we need to rename roles: `zookeeper_quorum` to `zookeeper`. If you have `zookeeper` then you fire up `ZooKeeperServerEmbedded`. And if not, then you default to `QuorumPeerMain`.
Thirdly, why is `ZkContainer` in `o.a.s.core` and not `o.a.s.cloud`?
I did take a stab at the `SolrZkServer` using the `ZooKeeperServerEmbedded`, updating the start/stop methods, but failed ;-(. Going to need to put this down for a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2740519696
##########
solr/core/src/test/org/apache/solr/cloud/TestEmbeddedZkQuorum.java:
##########
Review Comment:
I had to mark the `testQuorumLossAndRecovery` test case as AwaitsFix, as it never got stable. After killing 2 nodes an starting them again, we had data loss in the collection, even if it had three replicas. I did not check if the three replicas were distributed across the three nodes, but I assume that's the default.. But the test killing one node/zk seems to pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2434250136
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -192,9 +193,13 @@ public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
// we are ZooKeeper enabled
try {
// If this is an ensemble, allow for a long connect time for other servers to come up
- if (zkServerEnabled && zkServer.getServers().size() > 1) {
+ if (zkServerEnabled && zkServer != null && zkServer.getServers().size() > 1) {
zkClientConnectTimeout = 24 * 60 * 60 * 1000; // 1 day for embedded ensemble
log.info("Zookeeper client={} Waiting for a quorum.", zookeeperHost);
+ } else if (zkServerEnabled && runAsQuorum) {
+ // Quorum mode also needs long timeout for other nodes to start
+ zkClientConnectTimeout = 24 * 60 * 60 * 1000; // 1 day for embedded quorum
Review Comment:
Lovely. I think we were trying to put units in the variable names in other places ;-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-4418544846
@gerlowskija FYI - I updated the PR description to match current code state
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217600740
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
So I modified things so that it is now enough to specify the `zookeeper_quorum` node role, and it will then enable embedded zk without the need for also `solr.zookeeper.server.enabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217834548
##########
solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc:
##########
@@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x200000
====
======
+[[embedded-zookeeper-ensemble]]
+== Embedded ZooKeeper Ensemble (Experimental)
+
+WARNING: This feature is *experimental / alpha* and is not suitable for production use.
+The configuration interface may change without notice in future Solr releases.
+
+Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures.
+Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble).
+This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters.
+
+=== How It Works
+
+Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup.
+Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`.
+
+Port scheme used by each node:
+
+|===
+|Port |Formula |Example (Solr on 8983)
+
+|Solr HTTP
+|`solr_port`
+|8983
+
+|ZK client
+|`solr_port + 1000`
+|9983
+
+|ZK quorum peer
+|`zk_client_port + 1`
+|9984
+
+|ZK leader election
+|`zk_client_port + 2`
+|9985
+|===
+
+=== Starting a 3-Node Embedded ZK Ensemble
+
+The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000`), and every node must use `zookeeper_quorum:on`.
+All three nodes must be started before the quorum can elect a leader and become operational.
+
+NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions.
+
+[source,bash]
+----
+export LH="localhost"
+ZK="-z $LH:21000,$LH:31000,$LH:41000"
+ROLES="-Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on"
+
+bin/solr start -p 20000 $ZK $ROLES
+bin/solr start -p 30000 $ZK $ROLES
+bin/solr start -p 40000 $ZK $ROLES
Review Comment:
I fixed it it commit ccf6ec3add1d0ee583d25f6cf518e4030d1f04be - appending `-<port>` to the `zoo_home` folder name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3816458591
I created SOLR-18094 as jira for this issue (sub-task of the SIP JIRA).
Let's focus on getting this PR in shape for releasing the building blocks of the Zk code and the node-role. Target 10.1 as an experimental feature, can document it as such and solicit feedback from users.
I made Precommit run, will spend some time some day to try to do a more thorough review. Let's make review comments for all known rough edges and things lacking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2435545434
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
Review Comment:
Yea, what is even an "external" zk in the context of our test framework?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2435541338
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -75,32 +81,40 @@ public class ZkContainer {
public ZkContainer() {}
public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
- boolean zkRun = EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ final boolean zkServerEnabled =
+ EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ // TODO NOCOMMIT - understand when zkRun is set
+ boolean zkQuorumNode = false;
+ if (NodeRoles.MODE_ON.equals(cc.nodeRoles.getRoleMode(NodeRoles.Role.ZOOKEEPER_QUORUM))) {
+ zkQuorumNode = true;
+ log.info("Starting node in ZooKeeper Quorum role.");
+ }
- if (zkRun && config == null)
+ if (zkServerEnabled && config == null) {
throw new SolrException(
SolrException.ErrorCode.SERVER_ERROR,
"Cannot start Solr in cloud mode - no cloud config provided");
+ }
+
+ if (config == null) {
Review Comment:
Who knows? Tons of tests failed due to NPE, and the existing code on main has this check, which was removed in this PR...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3408707634
FYI, the `.bats` test worked a few times for me and then quit working ;-(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217152288
##########
solr/core/src/java/org/apache/solr/cloud/SolrZkServer.java:
##########
@@ -253,10 +263,10 @@ public static Properties getProperties(Path configPath) throws ConfigException {
// Given zkHost=localhost:1111,localhost:2222 this will inject
// server.0=localhost:1112:1113
// server.1=localhost:2223:2224
- public static void injectServers(Properties props, boolean zkRun, String zkHost) {
+ public static void injectServers(Properties props, String zkHost) {
// if clientPort not already set, use zkRun
- if (zkRun && props.getProperty("clientPort") == null) {
+ if (props.getProperty("clientPort") == null) {
// int portIdx = zkRun.lastIndexOf(':');
int portIdx = "".lastIndexOf(':');
Review Comment:
@epugh This line was modified by you and causes the below if() to be dead code. Remember what's going on here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-2040686534
FYI: https://cwiki.apache.org/confluence/display/SOLR/SIP-14+Embedded+Zookeeper for context on others' interest in moving this direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3215665335
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -340,6 +346,274 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = reservePortPairs(numServers);
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
+ System.setProperty("solr.security.manager.enabled", "false");
+ System.setProperty("solr.node.roles", "data:on,overseer:allowed,zookeeper_quorum:on");
+ System.setProperty("solr.test.sys.prop1", "propone");
+ System.setProperty("solr.test.sys.prop2", "proptwo");
+ System.setProperty("solr.zookeeper.client.timeout", "300000"); // 5 minutes
+
+ // Phase 2: Start all nodes in parallel
+ List<Callable<JettySolrRunner>> startups = new ArrayList<>(numServers);
+ for (int i = 0; i < numServers; i++) {
+ final int solrPort = ports[i];
+ final String nodeName = newNodeName();
+ startups.add(
+ () -> {
+ Path runnerPath = createInstancePath(nodeName);
+ Files.write(runnerPath.resolve("solr.xml"), solrXml.getBytes(StandardCharsets.UTF_8));
+
+ Properties nodeProps = new Properties();
+ nodeProps.setProperty("zkHost", this.zkHost);
+ nodeProps.setProperty("hostPort", String.valueOf(solrPort));
+
+ JettyConfig newConfig = JettyConfig.builder(jettyConfig).setPort(solrPort).build();
+
+ JettySolrRunner jetty =
+ !trackJettyMetrics
+ ? new JettySolrRunner(runnerPath.toString(), nodeProps, newConfig)
+ : new JettySolrRunnerWithMetrics(runnerPath.toString(), nodeProps, newConfig);
+
+ int zkPort = solrPort + 1000;
+ log.info("Starting {} on port {} with ZK on port {}", nodeName, solrPort, zkPort);
+ jetty.start();
+ log.info("Node {} started successfully", nodeName);
+
+ jettys.add(jetty);
+ synchronized (startupWait) {
+ startupWait.notifyAll();
+ }
+ return jetty;
+ });
+ }
+
+ final ExecutorService executorLauncher =
+ ExecutorUtil.newMDCAwareCachedThreadPool(new SolrNamedThreadFactory("jetty-launcher"));
+ Collection<Future<JettySolrRunner>> futures = executorLauncher.invokeAll(startups);
+ ExecutorUtil.shutdownAndAwaitTermination(executorLauncher);
+ Exception startupError =
+ checkForExceptions(
+ "Error starting up MiniSolrCloudCluster with embedded ZK quorum", futures);
+ if (startupError != null) {
+ try {
+ this.shutdown();
+ } catch (Throwable t) {
+ startupError.addSuppressed(t);
+ }
+ throw startupError;
+ }
+
+ log.info("All {} nodes started, waiting for quorum formation...", numServers);
+ Thread.sleep(10000); // Wait for ZK quorum to fully form
+
+ // Initialize ZK paths and security (if provided)
+ try (SolrZkClient zkClient =
+ new SolrZkClient.Builder()
+ .withUrl(this.zkHost)
+ .withTimeout(60000, TimeUnit.MILLISECONDS)
+ .build()) {
+ if (!zkClient.exists("/solr")) {
+ zkClient.makePath("/solr", true);
+ }
+
+ if (jettyConfig.sslConfig != null && jettyConfig.sslConfig.isSSLMode()) {
+ zkClient.makePath(
+ "/solr" + ZkStateReader.CLUSTER_PROPS,
+ "{'urlScheme':'https'}".getBytes(StandardCharsets.UTF_8),
+ true);
+ }
+ if (securityJson.isPresent()) {
+ zkClient.makePath(
+ "/solr/security.json", securityJson.get().getBytes(Charset.defaultCharset()), true);
+ }
+ }
+
+ solrClient = buildSolrClientForQuorum(this.zkHost);
+
+ if (numServers > 0) {
+ waitForAllNodes(numServers, 60);
+ }
+
+ log.info("Embedded ZK quorum cluster started successfully with {} nodes", numServers);
+ }
+
+ /**
+ * Reserves port pairs for embedded ZK quorum mode. For each node, we need both a Solr port and a
+ * ZK port (Solr port + 1000). This method ensures both ports in each pair are available before
+ * returning.
+ *
+ * <p>The method keeps all ServerSockets open during the search to prevent race conditions where
+ * another process might grab a port between our check and actual usage.
+ *
+ * @param numPairs the number of port pairs to reserve
+ * @return array of Solr ports (ZK ports are Solr port + 1000)
+ * @throws IOException if unable to find enough available port pairs
+ */
+ private int[] reservePortPairs(int numPairs) throws IOException {
+ List<ServerSocket> solrSockets = new ArrayList<>();
+ List<ServerSocket> zkSockets = new ArrayList<>();
+ int[] ports = new int[numPairs];
+
+ try {
+ int pairsFound = 0;
+ int maxAttempts = numPairs * 100; // Reasonable limit to avoid infinite loops
+ int attempts = 0;
+
+ while (pairsFound < numPairs && attempts < maxAttempts) {
+ attempts++;
+ ServerSocket solrSocket = null;
+ ServerSocket zkSocket = null;
+
+ try {
+ // Try to get a random available port for Solr
+ solrSocket = new ServerSocket(0);
+ int solrPort = solrSocket.getLocalPort();
+ int zkPort = solrPort + 1000;
+
+ // Check if ZK port would exceed the valid port range (0-65535)
+ if (zkPort > 65535) {
+ solrSocket.close();
+ continue; // Skip this port and try again
+ }
+
+ // Verify the corresponding ZK port is also available
+ zkSocket = new ServerSocket(zkPort);
+
+ // Both ports are available - keep the sockets and record the port
+ solrSockets.add(solrSocket);
+ zkSockets.add(zkSocket);
+ ports[pairsFound] = solrPort;
+ pairsFound++;
+
+ if (log.isDebugEnabled()) {
+ log.debug(
+ "Reserved port pair {}/{}: Solr={}, ZK={}", pairsFound, numPairs, solrPort, zkPort);
+ }
+
+ } catch (IOException | IllegalArgumentException e) {
+ // ZK port was not available or invalid, close sockets and try again
+ if (solrSocket != null) {
+ try {
+ solrSocket.close();
+ } catch (IOException ignored) {
+ }
+ }
+ if (zkSocket != null) {
+ try {
+ zkSocket.close();
+ } catch (IOException ignored) {
+ }
+ }
+ }
+ }
+
+ if (pairsFound < numPairs) {
+ throw new IOException(
+ "Unable to find " + numPairs + " available port pairs after " + attempts + " attempts");
+ }
+ return ports;
+
+ } finally {
+ // Close all sockets now that we've recorded the ports
+ // The ports will remain available for immediate reuse
+ for (ServerSocket socket : solrSockets) {
+ try {
+ socket.close();
+ } catch (IOException e) {
+ log.warn("Error closing Solr socket", e);
+ }
+ }
+ for (ServerSocket socket : zkSockets) {
+ try {
+ socket.close();
+ } catch (IOException e) {
+ log.warn("Error closing ZK socket", e);
+ }
+ }
+ }
+ }
+
+ /**
+ * Get the ZK connection string. Works for both standard mode (using zkServer) and quorum mode
+ * (using zkHost field).
+ *
+ * @return ZK connection string
+ */
+ private String getZkAddress() {
+ if (zkHost != null) {
+ return zkHost; // Quorum mode
+ }
+ return zkServer.getZkAddress(); // Standard mode
Review Comment:
`zkServer` is null in embedded-quorum mode, but `newSolrClient(String)` still calls `getZkServer().getZkAddress()` (see `newSolrClient` around line ~1079). That will throw NPE for any caller using `cluster.newSolrClient(...)` with quorum mode enabled. Update `newSolrClient` (and any other callers) to use `getZkAddress()` instead of `getZkServer()` and/or guard against `zkServer == null`.
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -389,6 +663,87 @@ public void waitForNode(JettySolrRunner jetty, int timeoutSeconds)
timeoutSeconds, TimeUnit.SECONDS, (o, n) -> n != null && n.contains(nodeName));
}
+ /**
+ * Wait for the expected number of live nodes in the cluster.
+ *
+ * @param expectedCount expected number of live nodes
+ * @param timeoutSeconds timeout in seconds
+ * @throws InterruptedException if interrupted while waiting
+ * @throws TimeoutException if the expected count is not reached within the timeout
+ */
+ public void waitForLiveNodes(int expectedCount, int timeoutSeconds)
+ throws InterruptedException, TimeoutException {
+ TimeOut timeout = new TimeOut(timeoutSeconds, TimeUnit.SECONDS, TimeSource.NANO_TIME);
+ while (!timeout.hasTimedOut()) {
+ long runningNodes = jettys.stream().filter(JettySolrRunner::isRunning).count();
+ if (runningNodes == expectedCount) {
+ log.info("Verified {} live nodes", runningNodes);
+ return;
+ }
+ Thread.sleep(200);
+ }
+ // Final check after timeout
+ long actualCount = jettys.stream().filter(JettySolrRunner::isRunning).count();
+ throw new TimeoutException(
+ "Live node count mismatch: expected " + expectedCount + " but got " + actualCount);
Review Comment:
This `waitForLiveNodes` implementation counts locally running Jetty processes, but the method name/usage implies ZooKeeper `live_nodes` (and callers use it to wait for node loss/join). Since `stopJettySolrRunner(...)` removes jettys from the list, this can return immediately without waiting for ZooKeeper state to update. Consider implementing this using `getZkStateReader().waitForLiveNodes(...)` (similar to `waitForJettyToStop`) and/or rename to reflect what it actually checks.
##########
solr/core/src/test/org/apache/solr/cloud/TestEmbeddedZkQuorum.java:
##########
@@ -0,0 +1,332 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.nio.file.Path;
+import java.util.Locale;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test embedded ZooKeeper running in quorum mode within Solr nodes.
+ *
+ * <p>This test verifies that:
+ *
+ * <ul>
+ * <li>Multiple Solr nodes can start with embedded ZK in quorum mode
+ * <li>The ZK quorum forms correctly
+ * <li>Collections can be created and used
+ * <li>Documents can be indexed and queried
+ * <li>All resources are properly closed on shutdown
+ * </ul>
+ */
+@SolrTestCaseJ4.SuppressSSL
+public class TestEmbeddedZkQuorum extends SolrCloudTestCase {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String COLLECTION_NAME = "test_quorum_collection";
+ private static final int NUM_NODES = 3;
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ // Disable ZooKeeper JMX to avoid MBean registration conflicts during beasting
Review Comment:
Typo in comment: "during beasting" should be "during testing" (or similar).
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -340,6 +346,274 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = reservePortPairs(numServers);
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
+ System.setProperty("solr.security.manager.enabled", "false");
+ System.setProperty("solr.node.roles", "data:on,overseer:allowed,zookeeper_quorum:on");
+ System.setProperty("solr.test.sys.prop1", "propone");
+ System.setProperty("solr.test.sys.prop2", "proptwo");
+ System.setProperty("solr.zookeeper.client.timeout", "300000"); // 5 minutes
+
+ // Phase 2: Start all nodes in parallel
+ List<Callable<JettySolrRunner>> startups = new ArrayList<>(numServers);
+ for (int i = 0; i < numServers; i++) {
+ final int solrPort = ports[i];
+ final String nodeName = newNodeName();
+ startups.add(
+ () -> {
+ Path runnerPath = createInstancePath(nodeName);
+ Files.write(runnerPath.resolve("solr.xml"), solrXml.getBytes(StandardCharsets.UTF_8));
+
+ Properties nodeProps = new Properties();
+ nodeProps.setProperty("zkHost", this.zkHost);
+ nodeProps.setProperty("hostPort", String.valueOf(solrPort));
+
+ JettyConfig newConfig = JettyConfig.builder(jettyConfig).setPort(solrPort).build();
+
+ JettySolrRunner jetty =
+ !trackJettyMetrics
+ ? new JettySolrRunner(runnerPath.toString(), nodeProps, newConfig)
+ : new JettySolrRunnerWithMetrics(runnerPath.toString(), nodeProps, newConfig);
+
+ int zkPort = solrPort + 1000;
+ log.info("Starting {} on port {} with ZK on port {}", nodeName, solrPort, zkPort);
+ jetty.start();
+ log.info("Node {} started successfully", nodeName);
+
+ jettys.add(jetty);
+ synchronized (startupWait) {
+ startupWait.notifyAll();
+ }
+ return jetty;
+ });
+ }
+
+ final ExecutorService executorLauncher =
+ ExecutorUtil.newMDCAwareCachedThreadPool(new SolrNamedThreadFactory("jetty-launcher"));
+ Collection<Future<JettySolrRunner>> futures = executorLauncher.invokeAll(startups);
+ ExecutorUtil.shutdownAndAwaitTermination(executorLauncher);
+ Exception startupError =
+ checkForExceptions(
+ "Error starting up MiniSolrCloudCluster with embedded ZK quorum", futures);
+ if (startupError != null) {
+ try {
+ this.shutdown();
+ } catch (Throwable t) {
+ startupError.addSuppressed(t);
+ }
+ throw startupError;
+ }
+
+ log.info("All {} nodes started, waiting for quorum formation...", numServers);
+ Thread.sleep(10000); // Wait for ZK quorum to fully form
+
+ // Initialize ZK paths and security (if provided)
+ try (SolrZkClient zkClient =
+ new SolrZkClient.Builder()
+ .withUrl(this.zkHost)
+ .withTimeout(60000, TimeUnit.MILLISECONDS)
+ .build()) {
+ if (!zkClient.exists("/solr")) {
+ zkClient.makePath("/solr", true);
+ }
+
+ if (jettyConfig.sslConfig != null && jettyConfig.sslConfig.isSSLMode()) {
+ zkClient.makePath(
+ "/solr" + ZkStateReader.CLUSTER_PROPS,
+ "{'urlScheme':'https'}".getBytes(StandardCharsets.UTF_8),
+ true);
+ }
+ if (securityJson.isPresent()) {
+ zkClient.makePath(
+ "/solr/security.json", securityJson.get().getBytes(Charset.defaultCharset()), true);
+ }
+ }
+
+ solrClient = buildSolrClientForQuorum(this.zkHost);
+
+ if (numServers > 0) {
+ waitForAllNodes(numServers, 60);
+ }
+
+ log.info("Embedded ZK quorum cluster started successfully with {} nodes", numServers);
+ }
+
+ /**
+ * Reserves port pairs for embedded ZK quorum mode. For each node, we need both a Solr port and a
+ * ZK port (Solr port + 1000). This method ensures both ports in each pair are available before
+ * returning.
+ *
+ * <p>The method keeps all ServerSockets open during the search to prevent race conditions where
+ * another process might grab a port between our check and actual usage.
+ *
Review Comment:
The Javadoc claims the method "keeps all ServerSockets open" to prevent races, but the `finally` block closes all sockets before the caller starts Jetty/ZK. Closing the sockets releases the ports, so another process/thread can still grab them before use. Either keep the sockets open until after the servers have bound (and close them later), or adjust the comment/approach (e.g., let Jetty/ZK bind to ephemeral ports and discover them).
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -340,6 +346,274 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = reservePortPairs(numServers);
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
+ System.setProperty("solr.security.manager.enabled", "false");
+ System.setProperty("solr.node.roles", "data:on,overseer:allowed,zookeeper_quorum:on");
+ System.setProperty("solr.test.sys.prop1", "propone");
+ System.setProperty("solr.test.sys.prop2", "proptwo");
+ System.setProperty("solr.zookeeper.client.timeout", "300000"); // 5 minutes
+
+ // Phase 2: Start all nodes in parallel
+ List<Callable<JettySolrRunner>> startups = new ArrayList<>(numServers);
+ for (int i = 0; i < numServers; i++) {
+ final int solrPort = ports[i];
+ final String nodeName = newNodeName();
+ startups.add(
+ () -> {
+ Path runnerPath = createInstancePath(nodeName);
+ Files.write(runnerPath.resolve("solr.xml"), solrXml.getBytes(StandardCharsets.UTF_8));
+
+ Properties nodeProps = new Properties();
+ nodeProps.setProperty("zkHost", this.zkHost);
+ nodeProps.setProperty("hostPort", String.valueOf(solrPort));
+
+ JettyConfig newConfig = JettyConfig.builder(jettyConfig).setPort(solrPort).build();
+
+ JettySolrRunner jetty =
+ !trackJettyMetrics
+ ? new JettySolrRunner(runnerPath.toString(), nodeProps, newConfig)
+ : new JettySolrRunnerWithMetrics(runnerPath.toString(), nodeProps, newConfig);
+
+ int zkPort = solrPort + 1000;
+ log.info("Starting {} on port {} with ZK on port {}", nodeName, solrPort, zkPort);
+ jetty.start();
+ log.info("Node {} started successfully", nodeName);
+
+ jettys.add(jetty);
+ synchronized (startupWait) {
+ startupWait.notifyAll();
+ }
+ return jetty;
+ });
+ }
+
+ final ExecutorService executorLauncher =
+ ExecutorUtil.newMDCAwareCachedThreadPool(new SolrNamedThreadFactory("jetty-launcher"));
+ Collection<Future<JettySolrRunner>> futures = executorLauncher.invokeAll(startups);
+ ExecutorUtil.shutdownAndAwaitTermination(executorLauncher);
+ Exception startupError =
+ checkForExceptions(
+ "Error starting up MiniSolrCloudCluster with embedded ZK quorum", futures);
+ if (startupError != null) {
+ try {
+ this.shutdown();
+ } catch (Throwable t) {
+ startupError.addSuppressed(t);
+ }
+ throw startupError;
+ }
+
+ log.info("All {} nodes started, waiting for quorum formation...", numServers);
+ Thread.sleep(10000); // Wait for ZK quorum to fully form
Review Comment:
The fixed `Thread.sleep(10000)` makes cluster startup timing-dependent and can be flaky on slow/loaded CI machines. Prefer waiting on a concrete condition (e.g., connect with retry to `zkHost`, or use `ZkStateReader.waitForLiveNodes` / ensemble-ready checks) instead of a hard sleep.
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -340,6 +346,274 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = reservePortPairs(numServers);
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
+ System.setProperty("solr.security.manager.enabled", "false");
+ System.setProperty("solr.node.roles", "data:on,overseer:allowed,zookeeper_quorum:on");
+ System.setProperty("solr.test.sys.prop1", "propone");
+ System.setProperty("solr.test.sys.prop2", "proptwo");
+ System.setProperty("solr.zookeeper.client.timeout", "300000"); // 5 minutes
+
Review Comment:
This constructor sets multiple global `System` properties (e.g., `solr.zookeeper.server.enabled`, `solr.security.manager.enabled`, `solr.node.roles`) but never restores the previous values on shutdown. Since MiniSolrCloudCluster is used across many tests in the same JVM, these properties can leak into subsequent tests and change their behavior. Capture previous values and restore/clear them in `shutdown()` (or avoid using global sysprops if possible).
##########
solr/core/src/test/org/apache/solr/cloud/TestEmbeddedZkQuorum.java:
##########
@@ -0,0 +1,332 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.nio.file.Path;
+import java.util.Locale;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test embedded ZooKeeper running in quorum mode within Solr nodes.
+ *
+ * <p>This test verifies that:
+ *
+ * <ul>
+ * <li>Multiple Solr nodes can start with embedded ZK in quorum mode
+ * <li>The ZK quorum forms correctly
+ * <li>Collections can be created and used
+ * <li>Documents can be indexed and queried
+ * <li>All resources are properly closed on shutdown
+ * </ul>
+ */
+@SolrTestCaseJ4.SuppressSSL
+public class TestEmbeddedZkQuorum extends SolrCloudTestCase {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String COLLECTION_NAME = "test_quorum_collection";
+ private static final int NUM_NODES = 3;
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ // Disable ZooKeeper JMX to avoid MBean registration conflicts during beasting
+ System.setProperty("zookeeper.jmx.log4j.disable", "true");
+
+ // Get path to a test config
+ Path configPath = TEST_PATH().resolve("collection1").resolve("conf");
+
+ // Configure cluster with 3 nodes, each running embedded ZK
+ cluster =
+ configureCluster(NUM_NODES).addConfig("conf1", configPath).withEmbeddedZkQuorum().build();
+ cluster.waitForAllNodes(60);
+ }
+
+ @Test
+ public void testBasicQuorumFunctionality()
+ throws IOException, InterruptedException, TimeoutException {
+ for (int i = 0; i < NUM_NODES; i++) {
+ JettySolrRunner node = cluster.getJettySolrRunner(i);
+ assertTrue("Node " + i + " should be running", node.isRunning());
+ assertNotNull("Node " + i + " should have a NodeName", node.getNodeName());
+ }
+ }
+
+ @Test
+ public void testCollectionIndexing() throws Exception {
+ try (CloudSolrClient client = cluster.getSolrClient(COLLECTION_NAME)) {
+ CollectionAdminRequest.Create createCmd =
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf1", 1, 3);
+ createCmd.process(client);
+ cluster.waitForActiveCollection(COLLECTION_NAME, 1, 3);
+
+ // Index some documents
+ for (int i = 0; i < 10; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField("id", i);
+ doc.addField("title_s", "Test Document " + i);
+ doc.addField("content_t", "This is test content for document " + i);
+ client.add(doc);
+ }
+ client.commit();
+
+ // Query the documents
+ SolrQuery query = new SolrQuery("*:*");
+ query.setRows(100);
+ QueryResponse response = client.query(query);
+ SolrDocumentList results = response.getResults();
+
+ // Verify results
+ assertEquals("Should have 10 documents", 10, results.getNumFound());
+
+ CollectionAdminRequest.Delete deleteCmd =
+ CollectionAdminRequest.deleteCollection(COLLECTION_NAME);
+ deleteCmd.process(client);
+ }
+ }
+
+ /**
+ * Tests ZK quorum resilience when a single node fails and recovers.
+ *
+ * <p>This test verifies that:
+ *
+ * <ul>
+ * <li>A 3-node ZK quorum can lose 1 node and maintain quorum (2/3)
+ * <li>The cluster continues to accept writes with 2 nodes
+ * <li>A failed node can rejoin the quorum using the same ports
+ * <li>All data is preserved after node recovery
+ * </ul>
+ *
+ * <p>This test creates its own private cluster to avoid interfering with other tests.
+ */
+ @Test
+ public void testQuorumResilienceWithNodeFailure() throws Exception {
+ final String collectionName = "quorum_resilience";
+ final int initialDocs = 5;
+ final int docsWhileDown = 5;
+ final int docsAfterRecovery = 5;
+
+ // Create a private cluster for this test
+ Path configPath = TEST_PATH().resolve("collection1").resolve("conf");
+ MiniSolrCloudCluster privateCluster =
+ configureCluster(NUM_NODES).addConfig("conf1", configPath).withEmbeddedZkQuorum().build();
+
+ try {
+ privateCluster.waitForAllNodes(60);
+
+ // Create collection with replica on each node
+ CollectionAdminRequest.createCollection(collectionName, "conf1", 1, 3)
+ .process(privateCluster.getSolrClient());
+ privateCluster.waitForActiveCollection(collectionName, 1, 3);
+
+ try (CloudSolrClient client = privateCluster.getSolrClient(collectionName)) {
+ // Index initial documents and verify
+ indexDocuments(client, 0, initialDocs, "initial");
+ privateCluster.waitForDocCount(
+ collectionName, initialDocs, "initial documents", 120, TimeUnit.SECONDS);
+
+ // Stop one node (quorum maintained with 2/3 nodes)
+ JettySolrRunner stoppedNode = privateCluster.getJettySolrRunner(2);
+ String stoppedNodeName = stoppedNode.getNodeName();
+ if (log.isInfoEnabled()) {
+ log.info("Stopping node to test quorum resilience: {}", stoppedNodeName);
+ }
+ privateCluster.stopJettySolrRunner(stoppedNode);
+
+ // Wait for ZK to detect node loss and verify cluster still operational
+ privateCluster.waitForLiveNodes(2, 120);
+ indexDocuments(client, initialDocs, docsWhileDown, "during_failure");
+ privateCluster.waitForDocCount(
+ collectionName,
+ initialDocs + docsWhileDown,
+ "documents while node down",
+ 120,
+ TimeUnit.SECONDS);
+ if (log.isInfoEnabled()) {
+ log.info("Starting node {} again and testing functionality", stoppedNodeName);
+ }
+
+ privateCluster.startJettySolrRunner(stoppedNode, true);
+ privateCluster.waitForNode(stoppedNode, 120);
+
+ // Wait for cluster to stabilize and verify all nodes running
+ privateCluster.waitForLiveNodes(3, 120);
+
+ // CRITICAL: Wait for collection to become active (replicas up, leader elected)
+ // before attempting to index documents
+ privateCluster.waitForActiveCollection(collectionName, 120, TimeUnit.SECONDS, 1, 3);
+
+ privateCluster.waitForDocCount(
+ collectionName,
+ initialDocs + docsWhileDown,
+ "documents after recovery",
+ 120,
+ TimeUnit.SECONDS);
+
+ // Verify full cluster functionality by adding more documents
+ indexDocuments(client, initialDocs + docsWhileDown, docsAfterRecovery, "after_recovery");
+ privateCluster.waitForDocCount(
+ collectionName,
+ initialDocs + docsWhileDown + docsAfterRecovery,
+ "all documents",
+ 120,
+ TimeUnit.SECONDS);
+ }
+ } finally {
+ CollectionAdminRequest.deleteCollection(collectionName)
+ .process(privateCluster.getSolrClient());
+ privateCluster.shutdown();
+ }
+ }
+
+ /**
+ * Tests ZK quorum loss and recovery when majority of nodes fail.
+ *
+ * <p>This test verifies that:
+ *
+ * <ul>
+ * <li>A 3-node ZK quorum loses quorum when 2 nodes are down (1/3 remaining)
+ * <li>The surviving node maintains its replica but cannot process updates without quorum
+ * <li>Both failed nodes can be restarted to restore quorum
+ * <li>The cluster becomes operational again (can query and index documents)
+ * <li>Note: After catastrophic failure, some replicas may need time or manual intervention to
+ * fully recover
+ * </ul>
+ *
+ * <p>This test creates its own private cluster to avoid interfering with other tests. <b>Hard to
+ * make this test pass</b>
+ */
+ @AwaitsFix(bugUrl = "https://example.com/foo")
Review Comment:
`@AwaitsFix` uses a placeholder URL (`https://example.com/foo`) and the Javadoc says the test is "Hard to make this test pass". In this codebase, `@AwaitsFix` links to a real Jira (see other tests). Please either provide the real SOLR issue URL explaining the failure mode, or remove/disable the test until it’s reliable (otherwise this adds a permanently flaky/ignored test).
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -70,42 +80,105 @@ public class ZkContainer {
public ZkContainer() {}
public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
- boolean zkRun = EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ // zkServerEnabled is set whenever in solrCloud mode ('-c') but no explicit zkHost/ZK_HOST is
+ // provided.
+ final boolean zkServerEnabled =
+ EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ boolean zkQuorumNode = false;
+ if (NodeRoles.MODE_ON.equals(cc.nodeRoles.getRoleMode(NodeRoles.Role.ZOOKEEPER_QUORUM))) {
+ zkQuorumNode = true;
+ log.info("Starting node in ZooKeeper Quorum role.");
+ }
- if (zkRun && config == null)
+ if (zkServerEnabled && config == null) {
throw new SolrException(
SolrException.ErrorCode.SERVER_ERROR,
"Cannot start Solr in cloud mode - no cloud config provided");
+ }
+
+ if (config == null) {
+ log.info("Solr is running in standalone mode");
+ return;
+ }
- if (config == null) return; // not in zk mode
+ final boolean runAsQuorum = config.getZkHost() != null && zkQuorumNode;
String zookeeperHost = config.getZkHost();
+ final var solrHome = cc.getSolrHome();
+ if (zkServerEnabled) {
+ if (!runAsQuorum) {
+ // Old school ZooKeeperServerMain being used under the covers.
+ zkServer =
+ SolrZkServer.createAndStart(config.getZkHost(), solrHome, config.getSolrHostPort());
+
+ // set client from server config if not already set
+ if (zookeeperHost == null) {
+ zookeeperHost = zkServer.getClientString();
+ }
+ } else {
+ // ZooKeeperServerEmbedded being used under the covers.
+ // Figure out where to put zoo-data
+ final var zkHomeDir = solrHome.resolve("zoo_home");
+ final var zkDataDir = zkHomeDir.resolve("data");
+
+ // Populate a zoo.cfg
+ final String zooCfgTemplate =
+ ""
+ + "tickTime=2000\n"
+ + "initLimit=10\n"
+ + "syncLimit=5\n"
+ + "dataDir=@@DATA_DIR@@\n"
+ + "4lw.commands.whitelist=mntr,conf,ruok\n"
+ + "admin.enableServer=false\n"
+ + "clientPort=@@ZK_CLIENT_PORT@@\n";
+
+ final int zkPort = config.getSolrHostPort() + 1000;
+ String zooCfgContents =
+ zooCfgTemplate
+ .replace("@@DATA_DIR@@", zkDataDir.toString())
+ .replace("@@ZK_CLIENT_PORT@@", String.valueOf(zkPort));
+ final String[] zkHosts = config.getZkHost().split(",");
+ int myId = -1;
+ final String targetConnStringSection = config.getHost() + ":" + zkPort;
+ if (log.isInfoEnabled()) {
+ log.info(
+ "Trying to match {} against zkHostString {} to determine myid",
+ targetConnStringSection,
+ config.getZkHost());
+ }
+ for (int i = 0; i < zkHosts.length; i++) {
+ final String host = zkHosts[i];
+ if (targetConnStringSection.equals(zkHosts[i])) {
+ myId = (i + 1);
+ }
+ final var hostComponents = host.split(":");
+ final var zkServer = hostComponents[0];
+ final var zkClientPort = Integer.valueOf(hostComponents[1]);
+ final var zkQuorumPort = zkClientPort - 4000;
+ final var zkLeaderPort = zkClientPort - 3000;
Review Comment:
The quorum/leader ports are derived by subtracting constants (`clientPort - 4000` / `- 3000`). This doesn’t match the port scheme used elsewhere in Solr’s embedded-ZK code (`SolrZkServerProps.injectServers` uses `clientPort+1` and `clientPort+2`) and can produce invalid/negative ports depending on the client port. This will prevent the ZooKeeper quorum from forming reliably. Use an explicit, consistent mapping for peer/election ports (and ensure those ports are available).
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -340,6 +346,274 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = reservePortPairs(numServers);
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
+ System.setProperty("solr.security.manager.enabled", "false");
+ System.setProperty("solr.node.roles", "data:on,overseer:allowed,zookeeper_quorum:on");
+ System.setProperty("solr.test.sys.prop1", "propone");
+ System.setProperty("solr.test.sys.prop2", "proptwo");
+ System.setProperty("solr.zookeeper.client.timeout", "300000"); // 5 minutes
+
+ // Phase 2: Start all nodes in parallel
+ List<Callable<JettySolrRunner>> startups = new ArrayList<>(numServers);
+ for (int i = 0; i < numServers; i++) {
+ final int solrPort = ports[i];
+ final String nodeName = newNodeName();
+ startups.add(
+ () -> {
+ Path runnerPath = createInstancePath(nodeName);
+ Files.write(runnerPath.resolve("solr.xml"), solrXml.getBytes(StandardCharsets.UTF_8));
+
+ Properties nodeProps = new Properties();
+ nodeProps.setProperty("zkHost", this.zkHost);
+ nodeProps.setProperty("hostPort", String.valueOf(solrPort));
+
+ JettyConfig newConfig = JettyConfig.builder(jettyConfig).setPort(solrPort).build();
+
+ JettySolrRunner jetty =
+ !trackJettyMetrics
+ ? new JettySolrRunner(runnerPath.toString(), nodeProps, newConfig)
+ : new JettySolrRunnerWithMetrics(runnerPath.toString(), nodeProps, newConfig);
+
+ int zkPort = solrPort + 1000;
+ log.info("Starting {} on port {} with ZK on port {}", nodeName, solrPort, zkPort);
+ jetty.start();
+ log.info("Node {} started successfully", nodeName);
+
+ jettys.add(jetty);
+ synchronized (startupWait) {
+ startupWait.notifyAll();
+ }
+ return jetty;
+ });
+ }
+
+ final ExecutorService executorLauncher =
+ ExecutorUtil.newMDCAwareCachedThreadPool(new SolrNamedThreadFactory("jetty-launcher"));
+ Collection<Future<JettySolrRunner>> futures = executorLauncher.invokeAll(startups);
+ ExecutorUtil.shutdownAndAwaitTermination(executorLauncher);
+ Exception startupError =
+ checkForExceptions(
+ "Error starting up MiniSolrCloudCluster with embedded ZK quorum", futures);
+ if (startupError != null) {
+ try {
+ this.shutdown();
+ } catch (Throwable t) {
+ startupError.addSuppressed(t);
+ }
+ throw startupError;
+ }
+
+ log.info("All {} nodes started, waiting for quorum formation...", numServers);
+ Thread.sleep(10000); // Wait for ZK quorum to fully form
+
+ // Initialize ZK paths and security (if provided)
+ try (SolrZkClient zkClient =
+ new SolrZkClient.Builder()
+ .withUrl(this.zkHost)
+ .withTimeout(60000, TimeUnit.MILLISECONDS)
+ .build()) {
+ if (!zkClient.exists("/solr")) {
+ zkClient.makePath("/solr", true);
+ }
+
+ if (jettyConfig.sslConfig != null && jettyConfig.sslConfig.isSSLMode()) {
+ zkClient.makePath(
+ "/solr" + ZkStateReader.CLUSTER_PROPS,
+ "{'urlScheme':'https'}".getBytes(StandardCharsets.UTF_8),
+ true);
+ }
+ if (securityJson.isPresent()) {
+ zkClient.makePath(
+ "/solr/security.json", securityJson.get().getBytes(Charset.defaultCharset()), true);
+ }
+ }
+
+ solrClient = buildSolrClientForQuorum(this.zkHost);
+
+ if (numServers > 0) {
+ waitForAllNodes(numServers, 60);
+ }
+
+ log.info("Embedded ZK quorum cluster started successfully with {} nodes", numServers);
+ }
+
+ /**
+ * Reserves port pairs for embedded ZK quorum mode. For each node, we need both a Solr port and a
+ * ZK port (Solr port + 1000). This method ensures both ports in each pair are available before
+ * returning.
+ *
+ * <p>The method keeps all ServerSockets open during the search to prevent race conditions where
+ * another process might grab a port between our check and actual usage.
+ *
+ * @param numPairs the number of port pairs to reserve
+ * @return array of Solr ports (ZK ports are Solr port + 1000)
+ * @throws IOException if unable to find enough available port pairs
+ */
+ private int[] reservePortPairs(int numPairs) throws IOException {
+ List<ServerSocket> solrSockets = new ArrayList<>();
+ List<ServerSocket> zkSockets = new ArrayList<>();
+ int[] ports = new int[numPairs];
+
+ try {
+ int pairsFound = 0;
+ int maxAttempts = numPairs * 100; // Reasonable limit to avoid infinite loops
+ int attempts = 0;
+
+ while (pairsFound < numPairs && attempts < maxAttempts) {
+ attempts++;
+ ServerSocket solrSocket = null;
+ ServerSocket zkSocket = null;
+
+ try {
+ // Try to get a random available port for Solr
+ solrSocket = new ServerSocket(0);
+ int solrPort = solrSocket.getLocalPort();
+ int zkPort = solrPort + 1000;
+
+ // Check if ZK port would exceed the valid port range (0-65535)
+ if (zkPort > 65535) {
+ solrSocket.close();
+ continue; // Skip this port and try again
+ }
+
+ // Verify the corresponding ZK port is also available
+ zkSocket = new ServerSocket(zkPort);
+
+ // Both ports are available - keep the sockets and record the port
+ solrSockets.add(solrSocket);
+ zkSockets.add(zkSocket);
+ ports[pairsFound] = solrPort;
+ pairsFound++;
+
+ if (log.isDebugEnabled()) {
+ log.debug(
+ "Reserved port pair {}/{}: Solr={}, ZK={}", pairsFound, numPairs, solrPort, zkPort);
+ }
+
+ } catch (IOException | IllegalArgumentException e) {
+ // ZK port was not available or invalid, close sockets and try again
+ if (solrSocket != null) {
+ try {
+ solrSocket.close();
+ } catch (IOException ignored) {
+ }
+ }
+ if (zkSocket != null) {
+ try {
+ zkSocket.close();
+ } catch (IOException ignored) {
+ }
+ }
Review Comment:
`reservePortPairs` only verifies availability of the Solr port and the ZooKeeper client port (`solrPort+1000`). A ZooKeeper quorum peer also needs additional ports for quorum/election traffic (typically `clientPort+1` and `clientPort+2`). If those ports are in use, the embedded ZK server can fail to bind even though this method succeeded; consider reserving/checking those ports too (or making the port scheme explicit).
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -70,42 +80,105 @@ public class ZkContainer {
public ZkContainer() {}
public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
- boolean zkRun = EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ // zkServerEnabled is set whenever in solrCloud mode ('-c') but no explicit zkHost/ZK_HOST is
+ // provided.
+ final boolean zkServerEnabled =
+ EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ boolean zkQuorumNode = false;
+ if (NodeRoles.MODE_ON.equals(cc.nodeRoles.getRoleMode(NodeRoles.Role.ZOOKEEPER_QUORUM))) {
+ zkQuorumNode = true;
+ log.info("Starting node in ZooKeeper Quorum role.");
+ }
- if (zkRun && config == null)
+ if (zkServerEnabled && config == null) {
throw new SolrException(
SolrException.ErrorCode.SERVER_ERROR,
"Cannot start Solr in cloud mode - no cloud config provided");
+ }
+
+ if (config == null) {
+ log.info("Solr is running in standalone mode");
+ return;
+ }
- if (config == null) return; // not in zk mode
+ final boolean runAsQuorum = config.getZkHost() != null && zkQuorumNode;
String zookeeperHost = config.getZkHost();
+ final var solrHome = cc.getSolrHome();
+ if (zkServerEnabled) {
+ if (!runAsQuorum) {
+ // Old school ZooKeeperServerMain being used under the covers.
+ zkServer =
+ SolrZkServer.createAndStart(config.getZkHost(), solrHome, config.getSolrHostPort());
+
+ // set client from server config if not already set
+ if (zookeeperHost == null) {
+ zookeeperHost = zkServer.getClientString();
+ }
+ } else {
+ // ZooKeeperServerEmbedded being used under the covers.
+ // Figure out where to put zoo-data
+ final var zkHomeDir = solrHome.resolve("zoo_home");
+ final var zkDataDir = zkHomeDir.resolve("data");
+
+ // Populate a zoo.cfg
+ final String zooCfgTemplate =
+ ""
+ + "tickTime=2000\n"
+ + "initLimit=10\n"
+ + "syncLimit=5\n"
+ + "dataDir=@@DATA_DIR@@\n"
+ + "4lw.commands.whitelist=mntr,conf,ruok\n"
+ + "admin.enableServer=false\n"
+ + "clientPort=@@ZK_CLIENT_PORT@@\n";
+
+ final int zkPort = config.getSolrHostPort() + 1000;
+ String zooCfgContents =
+ zooCfgTemplate
+ .replace("@@DATA_DIR@@", zkDataDir.toString())
+ .replace("@@ZK_CLIENT_PORT@@", String.valueOf(zkPort));
+ final String[] zkHosts = config.getZkHost().split(",");
+ int myId = -1;
+ final String targetConnStringSection = config.getHost() + ":" + zkPort;
+ if (log.isInfoEnabled()) {
+ log.info(
+ "Trying to match {} against zkHostString {} to determine myid",
+ targetConnStringSection,
+ config.getZkHost());
+ }
+ for (int i = 0; i < zkHosts.length; i++) {
+ final String host = zkHosts[i];
+ if (targetConnStringSection.equals(zkHosts[i])) {
+ myId = (i + 1);
+ }
+ final var hostComponents = host.split(":");
+ final var zkServer = hostComponents[0];
+ final var zkClientPort = Integer.valueOf(hostComponents[1]);
+ final var zkQuorumPort = zkClientPort - 4000;
+ final var zkLeaderPort = zkClientPort - 3000;
+ final String configEntry =
+ "server." + (i + 1) + "=" + zkServer + ":" + zkQuorumPort + ":" + zkLeaderPort + "\n";
+ zooCfgContents = zooCfgContents + configEntry;
+ }
+
+ if (myId == -1) {
+ throw new IllegalStateException(
+ "Unable to determine ZK 'myid' for target " + targetConnStringSection);
+ }
Review Comment:
`myId` detection requires an exact string match between `config.getHost()` and the host portion of `zkHost` entries. This will fail for common cases like `zkHost` using `localhost` while Solr’s configured host is `127.0.0.1`, causing startup to abort with `Unable to determine ZK 'myid'`. Consider matching by port (and/or resolving hostnames/loopback equivalence) instead of requiring exact host string equality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3241382915
##########
solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc:
##########
@@ -589,16 +589,17 @@ The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000
All three nodes must be started before the quorum can elect a leader and become operational.
NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions.
+When nodes share the same working directory, also set `solr.zookeeper.server.datadir` to a distinct path per node to avoid ZooKeeper data directory conflicts.
[source,bash]
----
export LH="localhost"
ZK_HOST="$LH:21000,$LH:31000,$LH:41000"
ROLES="data:on,overseer:allowed,zookeeper_quorum:on"
-bin/solr start -p 20000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES"
-bin/solr start -p 30000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES"
-bin/solr start -p 40000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES"
+bin/solr start -p 20000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -Dsolr.zookeeper.server.datadir=zoo_home_1
Review Comment:
is this really better? What if we did soemthing like `./blah/blah/zoo_home/localhost:21000` and `./blah/blah/zoo_home/localhost:31000` and ``./blah/blah/zoo_home/localhost:41000`, and generated it. I don't think the vast majority of folks even want to think about zook_home!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-4418795685
I addressed Copilot's review points, then ran another review with Claude and addressed some of those concerns.
Added ref-guide docs, marked as "experimental":
<img width="2322" height="1215" alt="Skjermbilde 2026-05-11 kl 10 21 03" src="https://github.com/user-attachments/assets/5361117b-adc6-4d34-ad63-6b63888ccc59" />
<img width="2041" height="280" alt="Skjermbilde 2026-05-11 kl 10 22 27" src="https://github.com/user-attachments/assets/7e493131-55df-4a28-9adb-983004c40d9a" />
<img width="2040" height="276" alt="Skjermbilde 2026-05-11 kl 10 22 07" src="https://github.com/user-attachments/assets/6ccf9d62-1424-4b55-84b3-421a230fa2d4" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217033659
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -389,6 +663,87 @@ public void waitForNode(JettySolrRunner jetty, int timeoutSeconds)
timeoutSeconds, TimeUnit.SECONDS, (o, n) -> n != null && n.contains(nodeName));
}
+ /**
+ * Wait for the expected number of live nodes in the cluster.
+ *
+ * @param expectedCount expected number of live nodes
+ * @param timeoutSeconds timeout in seconds
+ * @throws InterruptedException if interrupted while waiting
+ * @throws TimeoutException if the expected count is not reached within the timeout
+ */
+ public void waitForLiveNodes(int expectedCount, int timeoutSeconds)
+ throws InterruptedException, TimeoutException {
+ TimeOut timeout = new TimeOut(timeoutSeconds, TimeUnit.SECONDS, TimeSource.NANO_TIME);
+ while (!timeout.hasTimedOut()) {
+ long runningNodes = jettys.stream().filter(JettySolrRunner::isRunning).count();
+ if (runningNodes == expectedCount) {
+ log.info("Verified {} live nodes", runningNodes);
+ return;
+ }
+ Thread.sleep(200);
+ }
+ // Final check after timeout
+ long actualCount = jettys.stream().filter(JettySolrRunner::isRunning).count();
+ throw new TimeoutException(
+ "Live node count mismatch: expected " + expectedCount + " but got " + actualCount);
Review Comment:
Added a TODO to code..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217196903
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -70,42 +80,109 @@ public class ZkContainer {
public ZkContainer() {}
public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
- boolean zkRun = EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ // zkServerEnabled is set whenever in solrCloud mode ('-c') but no explicit zkHost/ZK_HOST is
+ // provided.
+ final boolean zkServerEnabled =
+ EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ boolean zkQuorumNode = false;
+ if (NodeRoles.MODE_ON.equals(cc.nodeRoles.getRoleMode(NodeRoles.Role.ZOOKEEPER_QUORUM))) {
+ zkQuorumNode = true;
+ log.info("Starting node in ZooKeeper Quorum role.");
+ }
- if (zkRun && config == null)
+ if (zkServerEnabled && config == null) {
throw new SolrException(
SolrException.ErrorCode.SERVER_ERROR,
"Cannot start Solr in cloud mode - no cloud config provided");
+ }
+
+ if (config == null) {
+ log.info("Solr is running in standalone mode");
+ return;
+ }
- if (config == null) return; // not in zk mode
+ final boolean runAsQuorum = config.getZkHost() != null && zkQuorumNode;
String zookeeperHost = config.getZkHost();
+ final var solrHome = cc.getSolrHome();
+ if (zkServerEnabled) {
+ if (!runAsQuorum) {
+ // Old school ZooKeeperServerMain being used under the covers.
+ zkServer =
+ SolrZkServer.createAndStart(config.getZkHost(), solrHome, config.getSolrHostPort());
+
+ // set client from server config if not already set
+ if (zookeeperHost == null) {
+ zookeeperHost = zkServer.getClientString();
+ }
+ } else {
+ // ZooKeeperServerEmbedded being used under the covers.
+ // Figure out where to put zoo-data
+ final var zkHomeDir = solrHome.resolve("zoo_home");
+ final var zkDataDir = zkHomeDir.resolve("data");
+
+ // Populate a zoo.cfg
+ final String zooCfgTemplate =
+ ""
+ + "tickTime=2000\n"
+ + "initLimit=10\n"
+ + "syncLimit=5\n"
+ + "dataDir=@@DATA_DIR@@\n"
+ + "4lw.commands.whitelist=mntr,conf,ruok\n"
+ + "admin.enableServer=false\n"
+ + "clientPort=@@ZK_CLIENT_PORT@@\n";
+
+ final int zkPort = config.getSolrHostPort() + 1000;
+ String zooCfgContents =
+ zooCfgTemplate
+ .replace("@@DATA_DIR@@", zkDataDir.toString())
+ .replace("@@ZK_CLIENT_PORT@@", String.valueOf(zkPort));
+ final String[] zkHosts = config.getZkHost().split(",");
+ int myId = -1;
+ // TODO: myId detection uses exact string matching between config.getHost() and the host
+ // portion of zkHost entries. This fails when zkHost uses "localhost" but Solr is
+ // configured with "127.0.0.1" (or vice versa). Consider resolving hostnames/IPs before
+ // matching, or matching by port alone (which is unique per node in a single-machine setup).
+ final String targetConnStringSection = config.getHost() + ":" + zkPort;
+ if (log.isInfoEnabled()) {
+ log.info(
+ "Trying to match {} against zkHostString {} to determine myid",
+ targetConnStringSection,
+ config.getZkHost());
+ }
+ for (int i = 0; i < zkHosts.length; i++) {
+ final String host = zkHosts[i];
Review Comment:
Risk of ArrayIndexOutOfBounds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217187756
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -60,7 +65,12 @@ public class ZkContainer {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
protected ZkController zkController;
+
+ // zkServer (and SolrZkServer) wrap a ZooKeeperServerMain if standalone mode, but in quorum we
+ // just use ZooKeeperServerEmbedded
+ // directly! Why? Can we use ZooKeeperServerEmbedded in one node directly instead?
private SolrZkServer zkServer;
+ private ZooKeeperServerEmbedded zkServerEmbedded;
Review Comment:
Must be volatile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2435581422
##########
solr/core/src/test/org/apache/solr/cloud/TestEmbeddedZkQuorum.java:
##########
Review Comment:
There used to be such a test in the first phase, but it kept failing, so it was removed 😂🤣
Good point though. With this foundation it should be possible to keep adding more exceptional tests. Hey, perahps even randomize running with and without qorum. But currently we requre >=3 nodes. We could evolve the quorum to use 1 zk node if <3 and use 3 zk nodes for >=3. Then if you kill a solr node, it should all work even if that node also has zk.
But I think there were some issues with stopping a `JettySolrRunner` in quorum mode and then restarting it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3291948982
Spent a few minutes pulling in the latest 'main'. Hoping to give it a bit more cleanup in the next few days if I can, but I've been hoping that for more than a year at this point (😬 ), so if anyone else is interested please feel free to move this forward!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2740635105
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -75,42 +85,119 @@ public class ZkContainer {
public ZkContainer() {}
public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
- boolean zkRun = EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ // zkServerEnabled is set whenever in solrCloud mode ('-c') but no explicit zkHost/ZK_HOST is
+ // provided.
+ final boolean zkServerEnabled =
+ EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ boolean zkQuorumNode = false;
+ if (NodeRoles.MODE_ON.equals(cc.nodeRoles.getRoleMode(NodeRoles.Role.ZOOKEEPER_QUORUM))) {
+ zkQuorumNode = true;
+ log.info("Starting node in ZooKeeper Quorum role.");
+ }
- if (zkRun && config == null)
+ if (zkServerEnabled && config == null) {
throw new SolrException(
SolrException.ErrorCode.SERVER_ERROR,
"Cannot start Solr in cloud mode - no cloud config provided");
+ }
+
+ if (config == null) {
+ log.info("Solr is running in standalone mode");
+ return;
+ }
- if (config == null) return; // not in zk mode
+ final boolean runAsQuorum = config.getZkHost() != null && zkQuorumNode;
String zookeeperHost = config.getZkHost();
+ final var solrHome = cc.getSolrHome();
+ if (zkServerEnabled) {
+ if (!runAsQuorum) {
+ // Old school ZooKeeperServerMain being used under the covers.
+ String zkDataHome =
+ EnvUtils.getProperty(
+ "solr.zookeeper.server.datadir", solrHome.resolve("zoo_data").toString());
+ String zkConfHome =
+ EnvUtils.getProperty("solr.zookeeper.server.confdir", solrHome.toString());
+ zkServer =
+ new SolrZkServer(
+ stripChroot(config.getZkHost()),
+ Path.of(zkDataHome),
+ zkConfHome,
+ config.getSolrHostPort());
+ zkServer.parseConfig();
+ zkServer.start();
+
+ // set client from server config if not already set
+ if (zookeeperHost == null) {
+ zookeeperHost = zkServer.getClientString();
+ }
+ // TODO - should this code go in SolrZkServer to augment or replace its current
+ // capabilities? Doing so
+ // would definitely keep ZkContainer cleaner...
Review Comment:
@gerlowskija I removed NOCOMMIT in this comment to unblock other precommit tests. Making this review comment to make sure we don't forget about it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2737258681
##########
solr/core/src/test/org/apache/solr/cloud/TestEmbeddedZkQuorum.java:
##########
Review Comment:
@epugh See last commit which adds two resilience tests to `TestEmbeddedZkQuorum`.
* `testQuorumResilienceWithNodeFailure` tests that we are operational when losing 1/3 nodes
* `testQuorumLossAndRecovery` tests that we cannot create a collection when losing 2/3 nodes, and that it recovers back to quorum again
I have not beasted them but pretty happy to see them pass in the first place. And as you probably guessed I did not write those tests, but Claude did and he needed some iterations, so there may absolutely be rough edges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217562546
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -70,42 +81,119 @@ public class ZkContainer {
public ZkContainer() {}
public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
- boolean zkRun = EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ // zkServerEnabled is set whenever in solrCloud mode ('-c') but no explicit zkHost/ZK_HOST is
+ // provided.
+ final boolean zkServerEnabled =
+ EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ boolean zkQuorumNode = false;
+ if (NodeRoles.MODE_ON.equals(cc.nodeRoles.getRoleMode(NodeRoles.Role.ZOOKEEPER_QUORUM))) {
+ zkQuorumNode = true;
+ log.info("Starting node in ZooKeeper Quorum role.");
+ }
- if (zkRun && config == null)
+ if (zkServerEnabled && config == null) {
throw new SolrException(
SolrException.ErrorCode.SERVER_ERROR,
"Cannot start Solr in cloud mode - no cloud config provided");
+ }
- if (config == null) return; // not in zk mode
+ if (config == null) {
+ log.info("Solr is running in standalone mode");
+ return;
+ }
+
+ final boolean runAsQuorum = config.getZkHost() != null && zkQuorumNode;
String zookeeperHost = config.getZkHost();
+ final var solrHome = cc.getSolrHome();
+ if (zkServerEnabled) {
+ if (!runAsQuorum) {
Review Comment:
It should be enough to specify the node role to start zk, no need for also `-dsolr.zookeeper.server.enabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217570947
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -241,9 +333,17 @@ public SolrMetricsContext getSolrMetricsContext() {
}
}
- private String stripChroot(String zkRun) {
- if (zkRun == null || zkRun.trim().isEmpty() || zkRun.lastIndexOf('/') < 0) return zkRun;
- return zkRun.substring(0, zkRun.lastIndexOf('/'));
+ private void startZooKeeperServerEmbedded(int port, String zkHomeDir) throws Exception {
+ Properties p = new Properties();
+ try (FileReader fr = new FileReader(zkHomeDir + "/zoo.cfg", StandardCharsets.UTF_8)) {
+ p.load(fr);
+ }
Review Comment:
Redundant read of zoo.cfg, the file is already in memory in call site. Refactor this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3229656915
##########
solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc:
##########
@@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x200000
====
======
+[[embedded-zookeeper-ensemble]]
+== Embedded ZooKeeper Ensemble (Experimental)
+
+WARNING: This feature is *experimental / alpha* and is not suitable for production use.
+The configuration interface may change without notice in future Solr releases.
+
+Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures.
+Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble).
+This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters.
+
+=== How It Works
+
+Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup.
+Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`.
+
+Port scheme used by each node:
+
+|===
+|Port |Formula |Example (Solr on 8983)
+
+|Solr HTTP
+|`solr_port`
+|8983
+
+|ZK client
+|`solr_port + 1000`
+|9983
+
+|ZK quorum peer
+|`zk_client_port + 1`
+|9984
+
+|ZK leader election
+|`zk_client_port + 2`
+|9985
Review Comment:
I’d prefer if Solr occupied a more narrow port space, say within 100 ports total, so you could start Solr on 8000, 8100, 8200 and each instance would have ports close. Instead of node 1 using 8000, 9000, 10000, 11000, while node 2 using 8100, 9100, 10100, 11100….
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3241365135
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -192,13 +196,62 @@ public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
e);
}
} else if (zkServerEnabled) {
- // Old school ZooKeeperServerMain being used under the covers (bin/solr -c without ZK_HOST).
- zkServer =
- SolrZkServer.createAndStart(config.getZkHost(), solrHome, config.getSolrHostPort());
+ try {
+ final Path zkDataHome =
+ Path.of(
+ EnvUtils.getProperty(
+ "solr.zookeeper.server.datadir", solrHome.resolve("zoo_data").toString()));
Review Comment:
There is NOT a strong pattern. I think it's personal preference. For me, I always find the text version easier to read than the `SOLR_ZOOKEEPER_SERVER_DETAILED` version ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3243570062
##########
solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc:
##########
@@ -589,16 +589,17 @@ The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000
All three nodes must be started before the quorum can elect a leader and become operational.
NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions.
+When nodes share the same working directory, also set `solr.zookeeper.server.datadir` to a distinct path per node to avoid ZooKeeper data directory conflicts.
[source,bash]
----
export LH="localhost"
ZK_HOST="$LH:21000,$LH:31000,$LH:41000"
ROLES="data:on,overseer:allowed,zookeeper_quorum:on"
-bin/solr start -p 20000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES"
-bin/solr start -p 30000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES"
-bin/solr start -p 40000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES"
+bin/solr start -p 20000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -Dsolr.zookeeper.server.datadir=zoo_home_1
Review Comment:
Yeah, I think the whole point of this embedded mode is that people don't need to think about it. I think if "I want to do it somewhere special" then that leads you more towards the "I will set up my own zk ensemble"... Just thinking from the perspective of a non Solr expert... This feature is really going to be a hit among the novices to Solr who don't want to think about Cloud architectures ;-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2736903009
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
+ System.setProperty("solr.security.manager.enabled", "false");
+ System.setProperty("solr.node.roles", "data:on,overseer:allowed,zookeeper_quorum:on");
+ System.setProperty("solr.test.sys.prop1", "propone");
+ System.setProperty("solr.test.sys.prop2", "proptwo");
+ System.setProperty("solr.zookeeper.client.timeout", "300000"); // 5 minutes
+
+ // Phase 2: Start all nodes in parallel
+ List<Callable<JettySolrRunner>> startups = new ArrayList<>(numServers);
+ for (int i = 0; i < numServers; i++) {
+ final int solrPort = ports[i];
+ final String nodeName = newNodeName();
+ startups.add(
+ () -> {
+ Path runnerPath = createInstancePath(nodeName);
+ Files.write(runnerPath.resolve("solr.xml"), solrXml.getBytes(StandardCharsets.UTF_8));
+
+ Properties nodeProps = new Properties();
+ nodeProps.setProperty("zkHost", this.zkHost);
+ nodeProps.setProperty("hostPort", String.valueOf(solrPort));
+
+ JettyConfig newConfig = JettyConfig.builder(jettyConfig).setPort(solrPort).build();
+
+ JettySolrRunner jetty =
+ !trackJettyMetrics
+ ? new JettySolrRunner(runnerPath.toString(), nodeProps, newConfig)
+ : new JettySolrRunnerWithMetrics(runnerPath.toString(), nodeProps, newConfig);
+
+ int zkPort = solrPort + 1000;
+ log.info("Starting {} on port {} with ZK on port {}", nodeName, solrPort, zkPort);
+ jetty.start();
+ log.info("Node {} started successfully", nodeName);
+
+ jettys.add(jetty);
+ synchronized (startupWait) {
+ startupWait.notifyAll();
+ }
+ return jetty;
+ });
+ }
+
+ final ExecutorService executorLauncher =
+ ExecutorUtil.newMDCAwareCachedThreadPool(new SolrNamedThreadFactory("jetty-launcher"));
+ Collection<Future<JettySolrRunner>> futures = executorLauncher.invokeAll(startups);
+ ExecutorUtil.shutdownAndAwaitTermination(executorLauncher);
+ Exception startupError =
+ checkForExceptions(
+ "Error starting up MiniSolrCloudCluster with embedded ZK quorum", futures);
+ if (startupError != null) {
+ try {
+ this.shutdown();
+ } catch (Throwable t) {
+ startupError.addSuppressed(t);
+ }
+ throw startupError;
+ }
+
+ log.info("All {} nodes started, waiting for quorum formation...", numServers);
+ Thread.sleep(10000); // Wait for ZK quorum to fully form
+
+ // Initialize ZK paths and security (if provided)
+ try (SolrZkClient zkClient =
+ new SolrZkClient.Builder()
+ .withUrl(this.zkHost)
+ .withTimeout(60000, TimeUnit.MILLISECONDS)
+ .build()) {
+ if (!zkClient.exists("/solr", true)) {
+ zkClient.makePath("/solr", false, true);
+ }
+ if (!zkClient.exists("/solr/initialized", true)) {
Review Comment:
It's not hallucination, but a way for the test to make sure that `urlScheme` and `security.json` are initialized properly in the cluster on first startup. As we only need one node to do this job, a dummy znode is written to zookeeper to say we are initialized.
This is test-code only so not critical what znodes we use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2736919896
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
Eric, what do you think about the `--quorum-hosts` / `SOLR_QUORUM_HOSTS` / `solr.quorum.hosts` setting? It will replace need for `-c` and `-z` and explicit node-role setting. When inspecting the "ps" command line of a solr node, you'll then clearly see that this cluster is started with the three given quorum nodes, and you immediately know it is a self-managed zk setup, as it lacks the explicit `-z` for an external cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3812367820
> > I wonder if the -c mode and the `Managed embedded quorum` are just the same? Maybe it's "the first three nodes that get fired up are all zk nodes".
>
> I don't like that kind of magic. I hope we can introduce this as an additional mode, then once it proves itself and is stable, we can make this mode the default when just starting a single node. But then we're at phase 4, where we have some degree of dynamic configuration and support for growing a quorum. I don't think that is important in the first few releases.
>
> I come to value stability and some degree of static non-mutable config. So it is not a crisis if you need to specify the `ZK_HOST` with all your quorum nodes when starting a solr cluster. It gives you clarify and stability. You don't get the cool dynamic stuff but in large clusters you setup quorum nodes once, either 3 or 5 and then don't touch them for years. All your 50 solr data nodes will have that exact same `ZK_HOST` string. And if you change it, well, you'll have a fun rolling restart of the entire cluster.
that makes sense... I suspect you have a lot more experience in running clusters at scale than I do!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2741464532
##########
solr/core/src/test/org/apache/solr/cloud/TestEmbeddedZkQuorum.java:
##########
Review Comment:
Would writing these as bats style change up the stability dynamics? And or help figure out why not working??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3480416124
> I did take a stab at the `SolrZkServer` using the `ZooKeeperServerEmbedded`, updating the start/stop methods, but failed ;-(. Going to need to put this down for a bit.
Thanks for chipping away on this. At some point we'll have something that can be released in 10.1 or 10.2 as an experimental feature for folks to try out, without dynamic reconfig. Then we can tackle the dynamic stuff...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2435354793
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -75,32 +81,40 @@ public class ZkContainer {
public ZkContainer() {}
public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
- boolean zkRun = EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ final boolean zkServerEnabled =
+ EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+ // TODO NOCOMMIT - understand when zkRun is set
+ boolean zkQuorumNode = false;
+ if (NodeRoles.MODE_ON.equals(cc.nodeRoles.getRoleMode(NodeRoles.Role.ZOOKEEPER_QUORUM))) {
+ zkQuorumNode = true;
+ log.info("Starting node in ZooKeeper Quorum role.");
+ }
- if (zkRun && config == null)
+ if (zkServerEnabled && config == null) {
throw new SolrException(
SolrException.ErrorCode.SERVER_ERROR,
"Cannot start Solr in cloud mode - no cloud config provided");
+ }
+
+ if (config == null) {
Review Comment:
How does this even get hit? Sometimes I get confused by all the null checks! If you are running in standalone, wouldn't this class never be run???!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2435561453
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -1174,16 +1379,33 @@ public MiniSolrCloudCluster build() throws Exception {
}
JettyConfig jettyConfig = jettyConfigBuilder.build();
- MiniSolrCloudCluster cluster =
- new MiniSolrCloudCluster(
- nodeCount,
- baseDir,
- solrXml,
- jettyConfig,
- null,
- securityJson,
- trackJettyMetrics,
- formatZkServer);
+ MiniSolrCloudCluster cluster;
+
+ if (useEmbeddedZkQuorum) {
Review Comment:
Hmm, this code **is** inside the builder :) So the builder chose to instantiate the MSCC using one of two constructors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3399599885
@gerlowskija I took a stab at getting in the use of the solr roles to determine if we hsould run embedded zk quorum mode, and it worked! Here is my start script, where each runs with `-f` so you need three terminals. Omit the `-f` and you don't need the seperate terminals.
```
mkdir -p ./three_nodes/node1/solr
mkdir -p ./three_nodes/node2/solr
mkdir -p ./three_nodes/node3/solr
SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8983 --solr-home "./three_nodes/node1/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on
SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8984 --solr-home "./three_nodes/node2/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on
SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8985 --solr-home "./three_nodes/node3/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3242921988
##########
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##########
@@ -192,13 +196,62 @@ public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
e);
}
} else if (zkServerEnabled) {
- // Old school ZooKeeperServerMain being used under the covers (bin/solr -c without ZK_HOST).
- zkServer =
- SolrZkServer.createAndStart(config.getZkHost(), solrHome, config.getSolrHostPort());
+ try {
+ final Path zkDataHome =
+ Path.of(
+ EnvUtils.getProperty(
+ "solr.zookeeper.server.datadir", solrHome.resolve("zoo_data").toString()));
Review Comment:
Yea, can be extracted to a constant if we start using the same property multiple times..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3241366574
##########
solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc:
##########
@@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x200000
====
======
+[[embedded-zookeeper-ensemble]]
+== Embedded ZooKeeper Ensemble (Experimental)
+
+WARNING: This feature is *experimental / alpha* and is not suitable for production use.
+The configuration interface may change without notice in future Solr releases.
+
+Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures.
+Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble).
+This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters.
+
+=== How It Works
+
+Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup.
+Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`.
+
+Port scheme used by each node:
+
+|===
+|Port |Formula |Example (Solr on 8983)
+
+|Solr HTTP
+|`solr_port`
+|8983
+
+|ZK client
+|`solr_port + 1000`
+|9983
+
+|ZK quorum peer
+|`zk_client_port + 1`
+|9984
+
+|ZK leader election
+|`zk_client_port + 2`
+|9985
Review Comment:
yeah...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2486414044
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
I wonder if `-z` should be reserved for connecting to an external ZK quorum, and we invent a new option for telling SolrCloud which solr nodes will run the zk node role? That would make it clearer to read, and we make ZK an implementation detail by not refering to it as ZK, but as `quorum-node`, e.g.
```bash
bin/solr start --quorum-nodes=solr-node-1:8983,solr-node-2:8983,solr-node-3:8983
```
That would mean we want Solr to manage our ZK, and translate into the correct `solr.zk.host` setting for each host, no matter if the host you are starting is one of the quorum nodes or not. And if the node you are starting is a quorum node, it will detect that it's own host name matches a quorum host, and add `-Dsolr.node.roles...` during startup, and the ZK bootstrap code will configure its internal ZK config using the `--quorum` / `-Dsolr.quorum.nodes` values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3409722429
With the help of Claude Code I added a JUnit test `TestEmbeddedZkQuorum`, that actually spins up a quorum cluster with `zookeeper_quorum:on` node roles (after securing a port so we know beforehand what the zkHost string will be).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2435548204
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
Sounds like node role should be enough, it should turn things on, perhaps set that property automagically?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3812120995
> I wonder if the -c mode and the `Managed embedded quorum` are just the same? Maybe it's "the first three nodes that get fired up are all zk nodes".
I don't like that kind of magic. I hope we can introduce this as an additional mode, then once it proves itself and is stable, we can make this mode the default when just starting a single node. But then we're at phase 4, where we have some degree of dynamic configuration and support for growing a quorum. I don't think that is important in the first few releases.
I come to value stability and some degree of static non-mutable config. So it is not a crisis if you need to specify the `ZK_HOST` with all your quorum nodes when starting a solr cluster. It gives you clarify and stability. You don't get the cool dynamic stuff but in large clusters you setup quorum nodes once, either 3 or 5 and then don't touch them for years. All your 50 solr data nodes will have that exact same `ZK_HOST` string. And if you change it, well, you'll have a fun rolling restart of the entire cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2737524063
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
Can I point you to https://github.com/apache/solr/pull/2783 where I added a page to the Ref Guide on "Deployment Topologies". I would LOVE your input, becasue I sort of guess how people would do things in a Kubernetes environment with solr-operator. I assumed that you would have one big pool, with some less resources nodes for `zookeeper_quorum:on,data:off,overseer:preferred ` and others with more resources for `zookeeper_quorum:off,data:on`. I think you are suggesting that isn't actually likely, they would be seperate.
Here is a diagram of some nodes doing both data and zookeeper:
<img width="1113" height="247" alt="image" src="https://github.com/user-attachments/assets/f7585f0c-a5b9-43d7-b8bc-53a9a302419f" />
and then what I have for Kubernetes ( but based on my very vague knowledge)
<img width="1134" height="647" alt="image" src="https://github.com/user-attachments/assets/a90c55a8-d6ef-4003-b9a8-a330520fd34f" />.
I'd love some PR updates on how from your expereince people actually would scale up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2736943031
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
I'm thinking ahead to solr-operator as well. When we have self managed zk, the operator should probably spin up two independent `StatefulSet`s, one "solr-quorum" with three low speced PODs with `zookeeper_quorum:on,data:off,overseer:preferred`. While the "solr-data" set of nodes will be today's set of data-nodes, automatically provisioned with a `ZK_HOST` pointing to the three quorum nodes. In k8s the host / service names will be stable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2435321334
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
I am a bit curious if we need this system property AND `zookeeper_quorum:on`? I wonder if the existence of `zookeeper_quorum:on` is enough? if this system property is what turns on zookeeper, I wish maybe it was `solr.mode=clustered` or something to specificy that it's the clustered (cloud) mode...
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
+ System.setProperty("solr.security.manager.enabled", "false");
+ System.setProperty("solr.node.roles", "data:on,overseer:allowed,zookeeper_quorum:on");
+ System.setProperty("solr.test.sys.prop1", "propone");
+ System.setProperty("solr.test.sys.prop2", "proptwo");
+ System.setProperty("solr.zookeeper.client.timeout", "300000"); // 5 minutes
+
+ // Phase 2: Start all nodes in parallel
+ List<Callable<JettySolrRunner>> startups = new ArrayList<>(numServers);
+ for (int i = 0; i < numServers; i++) {
+ final int solrPort = ports[i];
+ final String nodeName = newNodeName();
+ startups.add(
+ () -> {
+ Path runnerPath = createInstancePath(nodeName);
+ Files.write(runnerPath.resolve("solr.xml"), solrXml.getBytes(StandardCharsets.UTF_8));
+
+ Properties nodeProps = new Properties();
+ nodeProps.setProperty("zkHost", this.zkHost);
+ nodeProps.setProperty("hostPort", String.valueOf(solrPort));
+
+ JettyConfig newConfig = JettyConfig.builder(jettyConfig).setPort(solrPort).build();
+
+ JettySolrRunner jetty =
+ !trackJettyMetrics
+ ? new JettySolrRunner(runnerPath.toString(), nodeProps, newConfig)
+ : new JettySolrRunnerWithMetrics(runnerPath.toString(), nodeProps, newConfig);
+
+ int zkPort = solrPort + 1000;
+ log.info("Starting {} on port {} with ZK on port {}", nodeName, solrPort, zkPort);
+ jetty.start();
+ log.info("Node {} started successfully", nodeName);
+
+ jettys.add(jetty);
+ synchronized (startupWait) {
+ startupWait.notifyAll();
+ }
+ return jetty;
+ });
+ }
+
+ final ExecutorService executorLauncher =
+ ExecutorUtil.newMDCAwareCachedThreadPool(new SolrNamedThreadFactory("jetty-launcher"));
+ Collection<Future<JettySolrRunner>> futures = executorLauncher.invokeAll(startups);
+ ExecutorUtil.shutdownAndAwaitTermination(executorLauncher);
+ Exception startupError =
+ checkForExceptions(
+ "Error starting up MiniSolrCloudCluster with embedded ZK quorum", futures);
+ if (startupError != null) {
+ try {
+ this.shutdown();
+ } catch (Throwable t) {
+ startupError.addSuppressed(t);
+ }
+ throw startupError;
+ }
+
+ log.info("All {} nodes started, waiting for quorum formation...", numServers);
+ Thread.sleep(10000); // Wait for ZK quorum to fully form
+
+ // Initialize ZK paths and security (if provided)
+ try (SolrZkClient zkClient =
+ new SolrZkClient.Builder()
+ .withUrl(this.zkHost)
+ .withTimeout(60000, TimeUnit.MILLISECONDS)
+ .build()) {
+ if (!zkClient.exists("/solr", true)) {
+ zkClient.makePath("/solr", false, true);
+ }
+ if (!zkClient.exists("/solr/initialized", true)) {
Review Comment:
is this a concept in solr? this path? New to me!
##########
solr/core/src/test/org/apache/solr/cloud/TestEmbeddedZkQuorum.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.file.Path;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test embedded ZooKeeper running in quorum mode within Solr nodes.
+ *
+ * <p>This test verifies that:
+ *
+ * <ul>
+ * <li>Multiple Solr nodes can start with embedded ZK in quorum mode
+ * <li>The ZK quorum forms correctly
+ * <li>Collections can be created and used
+ * <li>Documents can be indexed and queried
+ * <li>All resources are properly closed on shutdown
+ * </ul>
+ */
+public class TestEmbeddedZkQuorum extends SolrCloudWithEmbeddedZkQuorumTestCase {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static final String COLLECTION_NAME = "test_quorum_collection";
+ private static final int NUM_NODES = 3;
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ // Get path to a test config
+ Path configPath = TEST_PATH().resolve("collection1").resolve("conf");
+
+ // Configure cluster with 3 nodes, each running embedded ZK
+ cluster =
+ configureClusterWithEmbeddedZkQuorum(NUM_NODES).addConfig("conf1", configPath).build();
+
+ log.info("Cluster configured with {} nodes", NUM_NODES);
+ }
+
+ @Test
+ public void testBasicQuorumFunctionality() throws Exception {
+ log.info("Starting testBasicQuorumFunctionality");
Review Comment:
I wonder if claude tools could go back and revamp old unit tests to be clearer?
##########
solr/core/src/test/org/apache/solr/cloud/TestEmbeddedZkQuorum.java:
##########
Review Comment:
This test is verifying the happy path! I think that we need a test that demonstrates that when you LOSE a node from a three node cluster, we still function, just degraded. I don't know if we have a test that verifies this behavior elsewhere (a stress test??) that could be reused with embedded zk?
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
Review Comment:
something about this being `true` seems odd... Should it be false since there isn't a externalZkServer?
##########
solr/test-framework/src/java/org/apache/solr/cloud/SolrCloudWithEmbeddedZkQuorumTestCase.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.lang.invoke.MethodHandles;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base class for SolrCloud tests that use embedded ZooKeeper running in quorum mode
Review Comment:
I really like the javadocs! as someone who doesn't live this java code everyday, they are helpful.
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -1174,16 +1379,33 @@ public MiniSolrCloudCluster build() throws Exception {
}
JettyConfig jettyConfig = jettyConfigBuilder.build();
- MiniSolrCloudCluster cluster =
- new MiniSolrCloudCluster(
- nodeCount,
- baseDir,
- solrXml,
- jettyConfig,
- null,
- securityJson,
- trackJettyMetrics,
- formatZkServer);
+ MiniSolrCloudCluster cluster;
+
+ if (useEmbeddedZkQuorum) {
Review Comment:
this big if statement seems challenging... Could a builder pattern make it easier? Thir are a lot of params and overlap with subtle differences ;-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2435550808
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
+ System.setProperty("solr.security.manager.enabled", "false");
+ System.setProperty("solr.node.roles", "data:on,overseer:allowed,zookeeper_quorum:on");
+ System.setProperty("solr.test.sys.prop1", "propone");
+ System.setProperty("solr.test.sys.prop2", "proptwo");
+ System.setProperty("solr.zookeeper.client.timeout", "300000"); // 5 minutes
+
+ // Phase 2: Start all nodes in parallel
+ List<Callable<JettySolrRunner>> startups = new ArrayList<>(numServers);
+ for (int i = 0; i < numServers; i++) {
+ final int solrPort = ports[i];
+ final String nodeName = newNodeName();
+ startups.add(
+ () -> {
+ Path runnerPath = createInstancePath(nodeName);
+ Files.write(runnerPath.resolve("solr.xml"), solrXml.getBytes(StandardCharsets.UTF_8));
+
+ Properties nodeProps = new Properties();
+ nodeProps.setProperty("zkHost", this.zkHost);
+ nodeProps.setProperty("hostPort", String.valueOf(solrPort));
+
+ JettyConfig newConfig = JettyConfig.builder(jettyConfig).setPort(solrPort).build();
+
+ JettySolrRunner jetty =
+ !trackJettyMetrics
+ ? new JettySolrRunner(runnerPath.toString(), nodeProps, newConfig)
+ : new JettySolrRunnerWithMetrics(runnerPath.toString(), nodeProps, newConfig);
+
+ int zkPort = solrPort + 1000;
+ log.info("Starting {} on port {} with ZK on port {}", nodeName, solrPort, zkPort);
+ jetty.start();
+ log.info("Node {} started successfully", nodeName);
+
+ jettys.add(jetty);
+ synchronized (startupWait) {
+ startupWait.notifyAll();
+ }
+ return jetty;
+ });
+ }
+
+ final ExecutorService executorLauncher =
+ ExecutorUtil.newMDCAwareCachedThreadPool(new SolrNamedThreadFactory("jetty-launcher"));
+ Collection<Future<JettySolrRunner>> futures = executorLauncher.invokeAll(startups);
+ ExecutorUtil.shutdownAndAwaitTermination(executorLauncher);
+ Exception startupError =
+ checkForExceptions(
+ "Error starting up MiniSolrCloudCluster with embedded ZK quorum", futures);
+ if (startupError != null) {
+ try {
+ this.shutdown();
+ } catch (Throwable t) {
+ startupError.addSuppressed(t);
+ }
+ throw startupError;
+ }
+
+ log.info("All {} nodes started, waiting for quorum formation...", numServers);
+ Thread.sleep(10000); // Wait for ZK quorum to fully form
+
+ // Initialize ZK paths and security (if provided)
+ try (SolrZkClient zkClient =
+ new SolrZkClient.Builder()
+ .withUrl(this.zkHost)
+ .withTimeout(60000, TimeUnit.MILLISECONDS)
+ .build()) {
+ if (!zkClient.exists("/solr", true)) {
+ zkClient.makePath("/solr", false, true);
+ }
+ if (!zkClient.exists("/solr/initialized", true)) {
Review Comment:
Could be some hallucination going on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3811087631
Thanks for taking a look at this @janhoy I think I've stalled a bit on this... It feels "so close' yet not sure how to get it over the hump.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3234355099
##########
solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc:
##########
@@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x200000
====
======
+[[embedded-zookeeper-ensemble]]
+== Embedded ZooKeeper Ensemble (Experimental)
+
+WARNING: This feature is *experimental / alpha* and is not suitable for production use.
+The configuration interface may change without notice in future Solr releases.
+
+Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures.
+Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble).
+This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters.
+
+=== How It Works
+
+Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup.
+Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`.
+
+Port scheme used by each node:
+
+|===
+|Port |Formula |Example (Solr on 8983)
+
+|Solr HTTP
+|`solr_port`
+|8983
+
+|ZK client
+|`solr_port + 1000`
+|9983
+
+|ZK quorum peer
+|`zk_client_port + 1`
+|9984
+
+|ZK leader election
+|`zk_client_port + 2`
+|9985
Review Comment:
And come to think of it, each and every of these ports should probably be configurable individually even if they default to some port offset...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3242919339
##########
solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc:
##########
@@ -589,16 +589,17 @@ The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000
All three nodes must be started before the quorum can elect a leader and become operational.
NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions.
+When nodes share the same working directory, also set `solr.zookeeper.server.datadir` to a distinct path per node to avoid ZooKeeper data directory conflicts.
[source,bash]
----
export LH="localhost"
ZK_HOST="$LH:21000,$LH:31000,$LH:41000"
ROLES="data:on,overseer:allowed,zookeeper_quorum:on"
-bin/solr start -p 20000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES"
-bin/solr start -p 30000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES"
-bin/solr start -p 40000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES"
+bin/solr start -p 20000 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -Dsolr.zookeeper.server.datadir=zoo_home_1
Review Comment:
The multi solr on single host is mostly for dev/test, so nice to have but not front and center? At least, now the zoo-data folder is configurable if folks want to place it somewhere special. But maybe we should also allow auto unique data folder through property? Also the zoo-config folder (defaulting to SOLR_HOME) needs the same...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-4453755547
> I just made the offsets for zookeeper server port and quorum and election ports configurable with sysprops. Still defaults to +1000. And defaulting quorum port to +1010 and election port to +1020. This means you can run 10 instances on same host on 8983, 8984, 8985, 8986 without any conflicts. And it is easily configurable to +1, +2, +3
I like it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217855127
##########
solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc:
##########
@@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x200000
====
======
+[[embedded-zookeeper-ensemble]]
+== Embedded ZooKeeper Ensemble (Experimental)
+
+WARNING: This feature is *experimental / alpha* and is not suitable for production use.
+The configuration interface may change without notice in future Solr releases.
+
+Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures.
+Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble).
+This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters.
+
+=== How It Works
+
+Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup.
+Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`.
+
+Port scheme used by each node:
+
+|===
+|Port |Formula |Example (Solr on 8983)
+
+|Solr HTTP
+|`solr_port`
+|8983
+
+|ZK client
+|`solr_port + 1000`
+|9983
+
+|ZK quorum peer
+|`zk_client_port + 1`
+|9984
+
+|ZK leader election
+|`zk_client_port + 2`
+|9985
Review Comment:
I modified the default ports used for quorum peer and leader election ports. They were something like client-port minus 3000 and minus 4000. Now they are +1 and +2. I feel we spread our port numbers too wide. But then after the change we got another problem that we could not start three test solr nodes on 8983, 8984, 8985, since the first one would occupy port 9983, 9984, 9985 for zookeeper.
I am willing to revert the +1, +2 to something else, but -3000 and -4000 seemed a bit arbitrary, why not +2000 and +3000 since client-port is +1000?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3228619564
##########
solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc:
##########
@@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x200000
====
======
+[[embedded-zookeeper-ensemble]]
+== Embedded ZooKeeper Ensemble (Experimental)
+
+WARNING: This feature is *experimental / alpha* and is not suitable for production use.
+The configuration interface may change without notice in future Solr releases.
+
+Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures.
+Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble).
+This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters.
+
+=== How It Works
+
+Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup.
+Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`.
+
+Port scheme used by each node:
+
+|===
+|Port |Formula |Example (Solr on 8983)
+
+|Solr HTTP
+|`solr_port`
+|8983
+
+|ZK client
+|`solr_port + 1000`
+|9983
+
+|ZK quorum peer
+|`zk_client_port + 1`
+|9984
+
+|ZK leader election
+|`zk_client_port + 2`
+|9985
Review Comment:
Yeah, after commenting elsewhere, I like your +2000 and +3000 idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217792431
##########
solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc:
##########
@@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x200000
====
======
+[[embedded-zookeeper-ensemble]]
+== Embedded ZooKeeper Ensemble (Experimental)
+
+WARNING: This feature is *experimental / alpha* and is not suitable for production use.
+The configuration interface may change without notice in future Solr releases.
+
+Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures.
+Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble).
+This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters.
+
+=== How It Works
+
+Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup.
+Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`.
+
+Port scheme used by each node:
+
+|===
+|Port |Formula |Example (Solr on 8983)
+
+|Solr HTTP
+|`solr_port`
+|8983
+
+|ZK client
+|`solr_port + 1000`
+|9983
+
+|ZK quorum peer
+|`zk_client_port + 1`
+|9984
+
+|ZK leader election
+|`zk_client_port + 2`
+|9985
+|===
+
+=== Starting a 3-Node Embedded ZK Ensemble
+
+The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000`), and every node must use `zookeeper_quorum:on`.
+All three nodes must be started before the quorum can elect a leader and become operational.
+
+NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions.
+
+[source,bash]
+----
+export LH="localhost"
+ZK="-z $LH:21000,$LH:31000,$LH:41000"
+ROLES="-Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on"
+
+bin/solr start -p 20000 $ZK $ROLES
+bin/solr start -p 30000 $ZK $ROLES
+bin/solr start -p 40000 $ZK $ROLES
Review Comment:
Running three nodes like this causes all three embedded zookeepers to use exact same `server/solr/zoo_home/data` folder, which is not good of course.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-2083599034
Great work. I think we should rip out the existing embedded ZK, which is a hack.
Do you want to proceed in a PR or perhaps in a central feature branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3408660185
* Merged in main branch
* Properly clean up the ZK server resource in close()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r2435767413
##########
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##########
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional<String> securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+ if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for embedded ZK quorum mode");
+ }
+ if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+ }
+
+ Objects.requireNonNull(securityJson);
+ this.baseDir = Objects.requireNonNull(baseDir);
+ this.jettyConfig = Objects.requireNonNull(jettyConfig);
+ this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+ this.trackJettyMetrics = trackJettyMetrics;
+ this.externalZkServer = true; // No ZkTestServer in quorum mode
+ this.zkServer = null; // No single ZK server
+
+ log.info("Starting cluster of {} servers with embedded ZK quorum in {}", numServers, baseDir);
+ Files.createDirectories(baseDir);
+
+ // Phase 1: Reserve random ports for all nodes
+ int[] ports = new int[numServers];
+ for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ ports[i] = socket.getLocalPort();
+ }
+ }
+
+ // Build the zkHost string with all ZK ports (Solr port + 1000)
+ StringBuilder zkHostBuilder = new StringBuilder();
+ for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+ zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+ }
+ this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+ if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers, java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+ }
+
+ // Set system properties for embedded ZK quorum mode
+ System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
If you dont have the node role, then you need it right? OR!!!????? here is an idea... What if you are starting a single solr node and want zk... The `bin/solr start` of SOlr 10, and was previoulsly `bin/solr start -c` in 9x... What if, instead, if you do NOT supply a `-Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on` or a `-z` property then we do that FOR you in the start up script. I believe that we ALREADY have logic that if you don't provide a `solr.node.roles` then you get `data:on,overseer:allowed` by default. See https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/core/NodeRoles.java#L32
Would that simplify things both internally and from a user perspective?
You either:
Start Solr old school with `bin/solr start --user-managed`
Or you start single node with embedded zk with `bin/solr start`
Or you start three nodes with embedded zk quorum with `bin/solr start -z blahblah:blahblah:blahblah -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on`
Or you start N nodes and connect to an external ZK with `bin/solr start -z external_zk:2181`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-4452794698
I just made the offsets for zookeeper server port and quorum and election ports configurable with sysprops. Still defaults to +1000. And defaulting quorum port to +1010 and election port to +1020. This means you can run 10 instances on same host on 8983, 8984, 8985, 8986 without any conflicts. And it is easily configurable to +1, +2, +3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3811895025
> Thanks for taking a look at this @janhoy I think I've stalled a bit on this... It feels "so close' yet not sure how to get it over the hump.
Yep, I got it up to speed with main, hardened the qourum test, and try to add test for stopping a quorum node and resume operation.
I think we need to focus on the startup modes / params to have a clear story for the three modes as you mentioned
* `--user-managed`
* `-c` (implicit) with one embedded ZK
* `-z external:2181` for existing external zk cluster
* Managed embedded quorum
For the managed embedded quorum, I believe we have proved that setting the node role explicitly and passing `-z` to all nodes pointing to those quorum nodes will do the trick. That is probably good enough for a first release?
## Phase 1:
Ship a experimental version with a working zk node-role, that starts a quorum that is somewhat stable.
## Phase 2:
Make it easier to setup for a small test cluster with some convenience flags (see above). integrate into solr operator.
Enable ZK ACL stuff by default.
## Phase 3:
Good support for SSL between solr and embedded zk quorum. Either self-provided certs, or self-signed certs during startup. Note, we can have non-ssl Solr API while still having SSL between solr and zookeeper.
## Phase 4:
Support dynamic reconfiguration. I don't know how important that really is. In k8s service-names are stable and you don't need to reconfigure zk really, except when changing the size of your quorum. But in self managed infrastructure, you may want to retire some servers and move zk over to another set of nodes by reconfiguring in two phases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Support running embedded-zk in "ensemble" mode [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-3812078622
I wonder if the -c mode and the `Managed embedded quorum` are just the same? Maybe it's "the first three nodes that get fired up are all zk nodes".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #2391:
URL: https://github.com/apache/solr/pull/2391#issuecomment-4416567928
Merged in latest main, hoping to pay some attention to this again.
Also kicked off a Copilot review, to have some early feedback.
Are you on board with trying to polish this PR branch in a "Phase 1" stable state which is mergable for 10.1? It would be a fantastic thing to get some alpha testers trying out, and then move on to Phase 2 in 10.2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217201302
##########
solr/packaging/build.gradle:
##########
@@ -305,6 +305,7 @@ task integrationTests(type: BatsTask) {
environment SOLR2_PORT: solrPort + 1
environment SOLR3_PORT: solrPort + 2
environment ZK_PORT: solrPort + 1000
+ environment SOLR_EXPORTER_PORT: solrPort + 100
Review Comment:
Unrelated change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3228599273
##########
solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc:
##########
@@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x200000
====
======
+[[embedded-zookeeper-ensemble]]
+== Embedded ZooKeeper Ensemble (Experimental)
+
+WARNING: This feature is *experimental / alpha* and is not suitable for production use.
+The configuration interface may change without notice in future Solr releases.
+
+Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures.
+Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble).
+This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters.
+
+=== How It Works
+
+Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup.
+Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`.
+
+Port scheme used by each node:
+
+|===
+|Port |Formula |Example (Solr on 8983)
+
+|Solr HTTP
+|`solr_port`
+|8983
+
+|ZK client
+|`solr_port + 1000`
+|9983
+
+|ZK quorum peer
+|`zk_client_port + 1`
+|9984
+
+|ZK leader election
+|`zk_client_port + 2`
+|9985
+|===
+
+=== Starting a 3-Node Embedded ZK Ensemble
+
+The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000`), and every node must use `zookeeper_quorum:on`.
+All three nodes must be started before the quorum can elect a leader and become operational.
+
+NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions.
+
+[source,bash]
+----
+export LH="localhost"
+ZK="-z $LH:21000,$LH:31000,$LH:41000"
+ROLES="-Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on"
+
+bin/solr start -p 20000 $ZK $ROLES
+bin/solr start -p 30000 $ZK $ROLES
+bin/solr start -p 40000 $ZK $ROLES
Review Comment:
so, we can't do 8983, 8984,8985? That is definitly a conceptual change for all of us. I'm used to zk being 1000 more that my port...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3228557730
##########
solr/core/src/java/org/apache/solr/cloud/SolrZkServer.java:
##########
@@ -253,10 +263,10 @@ public static Properties getProperties(Path configPath) throws ConfigException {
// Given zkHost=localhost:1111,localhost:2222 this will inject
// server.0=localhost:1112:1113
// server.1=localhost:2223:2224
- public static void injectServers(Properties props, boolean zkRun, String zkHost) {
+ public static void injectServers(Properties props, String zkHost) {
// if clientPort not already set, use zkRun
- if (zkRun && props.getProperty("clientPort") == null) {
+ if (props.getProperty("clientPort") == null) {
// int portIdx = zkRun.lastIndexOf(':');
int portIdx = "".lastIndexOf(':');
Review Comment:
not totally... And I think it was me hacking around a bit trying to figure out all the various sys props that existed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3229675779
##########
solr/core/src/java/org/apache/solr/cloud/SolrZkServer.java:
##########
@@ -253,10 +263,10 @@ public static Properties getProperties(Path configPath) throws ConfigException {
// Given zkHost=localhost:1111,localhost:2222 this will inject
// server.0=localhost:1112:1113
// server.1=localhost:2223:2224
- public static void injectServers(Properties props, boolean zkRun, String zkHost) {
+ public static void injectServers(Properties props, String zkHost) {
// if clientPort not already set, use zkRun
- if (zkRun && props.getProperty("clientPort") == null) {
+ if (props.getProperty("clientPort") == null) {
// int portIdx = zkRun.lastIndexOf(':');
int portIdx = "".lastIndexOf(':');
Review Comment:
Feel like figuring out what is the correct code here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org