You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/10/20 20:29:11 UTC
[GitHub] [lucene-solr] thelabdude opened a new pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
thelabdude opened a new pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010
# Description
See JIRA for description of the issue: https://issues.apache.org/jira/browse/SOLR-12182
# Solution
This PR takes the approach of storing the URL scheme (http or https) of the `base_url` for replicas in state.json using a replaceable parameter syntax `${scheme}://` instead of the actual scheme at the time of the write to ZK. The `${scheme}` var is replaced with the current value of the `urlScheme` when `state.json` is read from ZK. I even added some checking to heal existing state.json so we can avoid needing a migration tool. Although, that might be too subtle of a side-effect, so I'm happy to remove that check and just work with base_url's that start with `${scheme}://`
I initially tried setting a `ThreadLocal` that gives access to the `urlScheme` whenever we need to read these props from ZK. However, that ended up being problematic because we tend to read ZkNodeProps from ZK in many places. What I landed on is just setting a global var (which I don't love doing), but `urlScheme` really is a global variable that should be set once during initialization by reading from the cluster property stored in ZK. So I felt trying to treat this global as something that was highly dynamic made the code overly cumbersome.
I also tried to get rid of the `urlScheme` cluster property (re: https://issues.apache.org/jira/browse/SOLR-10202) but I'm not sure how SolrCloud client applications can resolve the correct `urlScheme` for the cluster without this property? On the server-side, sure we can just get the urlScheme from a Java System Property, but that won't be set for remote client applications that initialize via a connection to ZooKeeper. So I'm keeping the cluster property `urlScheme` for now.
I'm also not clear about the `urlScheme` Java System Property ... our tests seem to set it if using `https` and many test solr.xml refer to it.
# Tests
Existing tests cover these code changes. Re-enabled SSLMigrationTest
# Checklist
Please review the following and check all that apply:
- [x] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [x] I have created a Jira issue and added the issue ID to my pull request title.
- [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [x] I have developed this patch against the `master` branch.
- [x] I have run `./gradlew check`.
- [x] I have added tests for my changes.
- [ ] I have added documentation for the [Ref Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) (for Solr changes only).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] noblepaul edited a comment on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
noblepaul edited a comment on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-731661638
> we don't want to promise the community a zero-downtime upgrade to enable TLS, because it is a hard thing to promise
I don't have a problem if there is a downtime when you upgrade from one scheme to another. I only worry if the scheme remains same and a rolling restart is not possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] muse-dev[bot] commented on a change in pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
muse-dev[bot] commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r523248650
##########
File path: solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
##########
@@ -344,13 +344,8 @@ final private void doReplicateOnlyRecovery(SolrCore core) throws InterruptedExce
// though
try {
CloudDescriptor cloudDesc = this.coreDescriptor.getCloudDescriptor();
- ZkNodeProps leaderprops = zkStateReader.getLeaderRetry(
- cloudDesc.getCollectionName(), cloudDesc.getShardId());
- final String leaderBaseUrl = leaderprops.getStr(ZkStateReader.BASE_URL_PROP);
- final String leaderCoreName = leaderprops.getStr(ZkStateReader.CORE_NAME_PROP);
-
- String leaderUrl = ZkCoreNodeProps.getCoreUrl(leaderBaseUrl, leaderCoreName);
-
+ ZkNodeProps leaderprops = zkStateReader.getLeaderRetry(cloudDesc.getCollectionName(), cloudDesc.getShardId());
+ String leaderUrl = ZkCoreNodeProps.getCoreUrl(leaderprops);
String ourUrl = ZkCoreNodeProps.getCoreUrl(baseUrl, coreName);
boolean isLeader = leaderUrl.equals(ourUrl); // TODO: We can probably delete most of this code if we say this
Review comment:
*NULL_DEREFERENCE:* object `leaderUrl` last assigned on line 348 could be null and is dereferenced at line 351.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] broberson721 commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
broberson721 commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-996273618
Appreciate that.. Any sense how off 8.11.2/this fix would be? A couple months? Or much longer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] anshumg commented on a change in pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
anshumg commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r525809933
##########
File path: solr/core/src/test/org/apache/solr/cloud/LeaderElectionTest.java
##########
@@ -414,6 +425,11 @@ private void waitForLeader(List<ClientThread> threads, int seq)
private int getLeaderThread() throws KeeperException, InterruptedException {
String leaderUrl = getLeaderUrl("collection1", "shard1");
+ // strip off the scheme
+ final int at = leaderUrl.indexOf("://");
+ if (at != -1) {
+ leaderUrl = leaderUrl.substring(at+3);
Review comment:
Can you please add a space around `+`.
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -2131,12 +2128,7 @@ public LeaderElector getOverseerElector() {
static String generateNodeName(final String hostName,
final String hostPort,
final String hostContext) {
- try {
- return hostName + ':' + hostPort + '_' +
- URLEncoder.encode(trimLeadingAndTrailingSlashes(hostContext), "UTF-8");
- } catch (UnsupportedEncodingException e) {
- throw new Error("JVM Does not seem to support UTF-8", e);
- }
+ return hostName + ':' + hostPort + '_' + URLEncoder.encode(trimLeadingAndTrailingSlashes(hostContext), StandardCharsets.UTF_8);
Review comment:
Java8 doesn't have `URLEncoder.encode(String, Charset)`. Also, it'd through the exception on 8x.
I like it this way, just saying that you might have to remember that when merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on a change in pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r508867089
##########
File path: solr/core/src/test/org/apache/solr/cloud/SSLMigrationTest.java
##########
@@ -63,7 +61,7 @@ public void test() throws Exception {
testMigrateSSL(new SSLTestConfig(false, false));
}
- public void testMigrateSSL(SSLTestConfig sslConfig) throws Exception {
+ private void testMigrateSSL(SSLTestConfig sslConfig) throws Exception {
Review comment:
it's an internal test method and doesn't need to be public but I'll rename it to make it clear
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on a change in pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r508863404
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/GlobalStateVars.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public class GlobalStateVars {
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String SCHEME_VAR = "${scheme}://";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+
+ private static final GlobalStateVars _singleton = new GlobalStateVars();
+
+ public static GlobalStateVars singleton() {
+ return _singleton;
+ }
+
+ private String urlScheme = System.getProperty(URL_SCHEME, HTTP);
+
+ // no new! you have to use the static singleton!
+ private GlobalStateVars() {
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public void setUrlScheme(final String urlScheme) {
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ /**
+ * Given a URL with a replaceable parameter for the scheme, return a new URL with the correct scheme applied.
+ * @param url A URL to change the scheme (http|https)
+ * @return A new URL with the correct scheme or null if the supplied url remains unchanged.
+ */
+ public String applyUrlSchemeIfChanged(final String url) {
Review comment:
yeah, it's ugly for sure, but wanted to capture the fact that the url didn't change ... how about Optional instead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on a change in pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r508870225
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/GlobalStateVars.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public class GlobalStateVars {
Review comment:
I agree, thanks for confirming ;-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on a change in pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r508867169
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/GlobalStateVars.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public class GlobalStateVars {
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String SCHEME_VAR = "${scheme}://";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+
+ private static final GlobalStateVars _singleton = new GlobalStateVars();
Review comment:
ah yes, good call, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on a change in pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r523117430
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -1401,8 +1420,7 @@ public ZkCoreNodeProps getLeaderProps(final String collection,
byte[] data = zkClient.getData(
ZkStateReader.getShardLeadersPath(collection, slice), null, null,
true);
- ZkCoreNodeProps leaderProps = new ZkCoreNodeProps(
- ZkNodeProps.load(data));
+ ZkCoreNodeProps leaderProps = new ZkCoreNodeProps(ZkNodeProps.load(data));
Review comment:
that'll teach me to fix weird whitespacing!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-722639840
I think this is ready for review again ... still have to create CHANGES.txt entry and update the ref guide but let's get through another round of reviews, esp. around the approach I'm taking here. Once we're in agreement on the overall design, I'll update the docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-713171759
Thanks for the quick turn-around with a review @madrob ... will update CHANGES.txt and ref guide next
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-996275681
Can't really speak to a release date for 8.11.2 but bug fix releases can happen pretty quickly. On the other hand, it's better to let 8.11.1 live in the wild for a little bit before rushing out another bug fix release. If there's not traction on that from others in the community in a few weeks, I'll get the release going myself.
I've already been looking into a related issue, so will address this here as well: https://issues.apache.org/jira/browse/SOLR-15587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
noblepaul commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-731661638
> we don't want to promise the community a zero-downtime upgrade to enable TLS, because it is a hard thing to promise
I don't have a problem if there is a downtime when you upgrade from one schem to another. I only worry if the scheme remains same and a rolling restart is not posisble.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude merged pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
thelabdude merged pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on a change in pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
madrob commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r508842858
##########
File path: solr/core/src/test/org/apache/solr/cloud/SSLMigrationTest.java
##########
@@ -63,7 +61,7 @@ public void test() throws Exception {
testMigrateSSL(new SSLTestConfig(false, false));
}
- public void testMigrateSSL(SSLTestConfig sslConfig) throws Exception {
+ private void testMigrateSSL(SSLTestConfig sslConfig) throws Exception {
Review comment:
?
##########
File path: solr/core/src/java/org/apache/solr/core/ZkContainer.java
##########
@@ -263,4 +275,27 @@ public void close() {
public ExecutorService getCoreZkRegisterExecutorService() {
return coreZkRegister;
}
+
+ private void setGlobalUrlScheme(final ZkController zkCtrl) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String urlScheme = zkCtrl.getZkStateReader().getClusterProperty(ZkStateReader.URL_SCHEME, null);
+ if (StringUtils.isNotEmpty(urlScheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ GlobalStateVars.singleton().setUrlScheme(urlScheme);
+ } else {
+ final String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ // it's OK to set the cluster prop to https b/c we're making the cluster more secure, not less
+ // wouldn't do this for http though
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. Updating the cluster property to match.");
Review comment:
If we have an initially HTTP cluster, and then one node comes up with HTTPS, then that will bring down the rest of the cluster, right? I don't think we can do this rolling, however.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/GlobalStateVars.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public class GlobalStateVars {
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String SCHEME_VAR = "${scheme}://";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+
+ private static final GlobalStateVars _singleton = new GlobalStateVars();
Review comment:
I prefer the enum singleton pattern
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/GlobalStateVars.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public class GlobalStateVars {
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String SCHEME_VAR = "${scheme}://";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+
+ private static final GlobalStateVars _singleton = new GlobalStateVars();
+
+ public static GlobalStateVars singleton() {
+ return _singleton;
+ }
+
+ private String urlScheme = System.getProperty(URL_SCHEME, HTTP);
+
+ // no new! you have to use the static singleton!
+ private GlobalStateVars() {
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public void setUrlScheme(final String urlScheme) {
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ /**
+ * Given a URL with a replaceable parameter for the scheme, return a new URL with the correct scheme applied.
+ * @param url A URL to change the scheme (http|https)
+ * @return A new URL with the correct scheme or null if the supplied url remains unchanged.
+ */
+ public String applyUrlSchemeIfChanged(final String url) {
Review comment:
null on unchanged seems like a trap to me
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/ZkCoreNodeProps.java
##########
@@ -20,6 +20,9 @@
private ZkNodeProps nodeProps;
public ZkCoreNodeProps(ZkNodeProps nodeProps) {
+ if (nodeProps == null) {
Review comment:
Use Objects.requireNonNull
##########
File path: solr/core/src/java/org/apache/solr/core/ZkContainer.java
##########
@@ -31,6 +31,7 @@
import java.util.function.Supplier;
import org.apache.commons.lang3.StringUtils;
+import org.apache.solr.common.cloud.GlobalStateVars;
Review comment:
nit: alphabetize imports
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/ZkNodeProps.java
##########
@@ -105,6 +111,17 @@ public static ZkNodeProps load(byte[] bytes) {
@Override
public void write(JSONWriter jsonWriter) {
+ final String baseUrl = (String)propMap.get(ZkStateReader.BASE_URL_PROP);
+ if (baseUrl != null && !baseUrl.startsWith(SCHEME_VAR)) {
+ final int at = baseUrl.indexOf("://");
+ if (at != -1) {
+ final String updatedUrl = SCHEME_VAR + baseUrl.substring(at+3);
+ Map<String,Object> modMap = new LinkedHashMap<>(propMap);
+ modMap.put(ZkStateReader.BASE_URL_PROP, updatedUrl);
+ jsonWriter.write(modMap);
Review comment:
I think it's clearer here if we only have a single return from the method, so we should fall through or have an else.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/GlobalStateVars.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public class GlobalStateVars {
Review comment:
I don't think we want to encourage further use of this pattern. Let's call it something more specific like `UrlSchemeState`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] muse-dev[bot] commented on a change in pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
muse-dev[bot] commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r518357932
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -1401,8 +1420,7 @@ public ZkCoreNodeProps getLeaderProps(final String collection,
byte[] data = zkClient.getData(
ZkStateReader.getShardLeadersPath(collection, slice), null, null,
true);
- ZkCoreNodeProps leaderProps = new ZkCoreNodeProps(
- ZkNodeProps.load(data));
+ ZkCoreNodeProps leaderProps = new ZkCoreNodeProps(ZkNodeProps.load(data));
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `ZkController.getLeaderProps(...)` indirectly writes to field `noggit.JSONParser.devNull.buf` outside of synchronization.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-713573260
@murblanc right, using `${scheme}://` doesn't matter at all except for making it clear that gets replaced at runtime, at least to me, it makes it clear that is a variable that gets replaced. If the consensus is we don't want it there, that's fine too, as it really doesn't matter for the impl.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] muse-dev[bot] commented on a change in pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
muse-dev[bot] commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r517612808
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/ZkCoreNodeProps.java
##########
@@ -16,15 +16,20 @@
*/
package org.apache.solr.common.cloud;
+import java.util.Objects;
+
public class ZkCoreNodeProps {
private ZkNodeProps nodeProps;
public ZkCoreNodeProps(ZkNodeProps nodeProps) {
+ Objects.requireNonNull(nodeProps, "nodeProps should not be null");
this.nodeProps = nodeProps;
}
public String getCoreUrl() {
- return getCoreUrl(nodeProps.getStr(ZkStateReader.BASE_URL_PROP), nodeProps.getStr(ZkStateReader.CORE_NAME_PROP));
+ String baseUrl = nodeProps.getStr(ZkStateReader.BASE_URL_PROP);
+ Objects.requireNonNull(baseUrl, "No base_url in ZkNodeProps! "+nodeProps.toString());
+ return getCoreUrl(baseUrl, nodeProps.getStr(ZkStateReader.CORE_NAME_PROP));
Review comment:
*NULL_DEREFERENCE:* object `baseUrl` last assigned on line 30 could be null and is dereferenced by call to `getCoreUrl(...)` at line 32.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private String urlScheme = HTTP;
+ private boolean useLiveNodesUrlScheme = false;
+ private SortedSet<String> liveNodes = null;
+ private SolrZkClient zkClient = null;
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public synchronized void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public synchronized void setUrlScheme(final String urlScheme) {
+ if (!this.urlScheme.equals(urlScheme) && this.zkClient != null) {
+ throw new IllegalStateException("Global, immutable 'urlScheme' already set for this node!");
+ }
+
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ public boolean useLiveNodesUrlScheme() {
+ return useLiveNodesUrlScheme;
+ }
+
+ public String getBaseUrlForNodeName(String nodeName) {
+ Objects.requireNonNull(nodeName,"node_name should not be null");
+ if (nodeName.indexOf('_') == -1) {
+ nodeName += '_'; // underscore required to indicate context
+ }
+ return Utils.getBaseUrlForNodeName(nodeName, getUrlSchemeForNodeName(nodeName));
+ }
+
+ public String getUrlSchemeForNodeName(final String nodeName) {
+ return useLiveNodesUrlScheme ? getSchemeFromLiveNode(nodeName).orElse(urlScheme) : urlScheme;
+ }
+
+ /**
+ * Given a URL with a replaceable parameter for the scheme, return a new URL with the correct scheme applied.
+ * @param url A URL to change the scheme (http|https)
+ * @return A new URL with the correct scheme
+ */
+ public String applyUrlScheme(final String url) {
+ Objects.requireNonNull(url, "URL must not be null!");
+
+ String updatedUrl;
+ if (useLiveNodesUrlScheme && liveNodes != null) {
+ updatedUrl = applyUrlSchemeFromLiveNodes(url);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `UrlScheme.applyUrlScheme(...)` indirectly reads without synchronization from `this.zkClient`. Potentially races with write in method `UrlScheme.reset()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private String urlScheme = HTTP;
+ private boolean useLiveNodesUrlScheme = false;
+ private SortedSet<String> liveNodes = null;
+ private SolrZkClient zkClient = null;
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public synchronized void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public synchronized void setUrlScheme(final String urlScheme) {
+ if (!this.urlScheme.equals(urlScheme) && this.zkClient != null) {
+ throw new IllegalStateException("Global, immutable 'urlScheme' already set for this node!");
+ }
+
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ public boolean useLiveNodesUrlScheme() {
+ return useLiveNodesUrlScheme;
+ }
+
+ public String getBaseUrlForNodeName(String nodeName) {
+ Objects.requireNonNull(nodeName,"node_name should not be null");
+ if (nodeName.indexOf('_') == -1) {
+ nodeName += '_'; // underscore required to indicate context
+ }
+ return Utils.getBaseUrlForNodeName(nodeName, getUrlSchemeForNodeName(nodeName));
+ }
+
+ public String getUrlSchemeForNodeName(final String nodeName) {
+ return useLiveNodesUrlScheme ? getSchemeFromLiveNode(nodeName).orElse(urlScheme) : urlScheme;
+ }
+
+ /**
+ * Given a URL with a replaceable parameter for the scheme, return a new URL with the correct scheme applied.
+ * @param url A URL to change the scheme (http|https)
+ * @return A new URL with the correct scheme
+ */
+ public String applyUrlScheme(final String url) {
+ Objects.requireNonNull(url, "URL must not be null!");
+
+ String updatedUrl;
+ if (useLiveNodesUrlScheme && liveNodes != null) {
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `UrlScheme.applyUrlScheme(...)` reads without synchronization from `this.liveNodes`. Potentially races with write in method `UrlScheme.onChange(...)`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private String urlScheme = HTTP;
+ private boolean useLiveNodesUrlScheme = false;
+ private SortedSet<String> liveNodes = null;
+ private SolrZkClient zkClient = null;
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public synchronized void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public synchronized void setUrlScheme(final String urlScheme) {
+ if (!this.urlScheme.equals(urlScheme) && this.zkClient != null) {
+ throw new IllegalStateException("Global, immutable 'urlScheme' already set for this node!");
+ }
+
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ public boolean useLiveNodesUrlScheme() {
+ return useLiveNodesUrlScheme;
+ }
+
+ public String getBaseUrlForNodeName(String nodeName) {
+ Objects.requireNonNull(nodeName,"node_name should not be null");
+ if (nodeName.indexOf('_') == -1) {
+ nodeName += '_'; // underscore required to indicate context
+ }
+ return Utils.getBaseUrlForNodeName(nodeName, getUrlSchemeForNodeName(nodeName));
+ }
+
+ public String getUrlSchemeForNodeName(final String nodeName) {
+ return useLiveNodesUrlScheme ? getSchemeFromLiveNode(nodeName).orElse(urlScheme) : urlScheme;
+ }
+
+ /**
+ * Given a URL with a replaceable parameter for the scheme, return a new URL with the correct scheme applied.
+ * @param url A URL to change the scheme (http|https)
+ * @return A new URL with the correct scheme
+ */
+ public String applyUrlScheme(final String url) {
+ Objects.requireNonNull(url, "URL must not be null!");
+
+ String updatedUrl;
+ if (useLiveNodesUrlScheme && liveNodes != null) {
+ updatedUrl = applyUrlSchemeFromLiveNodes(url);
+ if (updatedUrl != null) {
+ return updatedUrl;
+ }
+ }
+
+ // heal an incorrect scheme if needed, otherwise return null indicating no change
+ final int at = url.indexOf("://");
+ return (at == -1) ? (urlScheme + "://" + url) : urlScheme + url.substring(at);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `UrlScheme.applyUrlScheme(...)` reads without synchronization from `this.urlScheme`. Potentially races with write in method `UrlScheme.reset()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private String urlScheme = HTTP;
+ private boolean useLiveNodesUrlScheme = false;
+ private SortedSet<String> liveNodes = null;
+ private SolrZkClient zkClient = null;
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public synchronized void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public synchronized void setUrlScheme(final String urlScheme) {
+ if (!this.urlScheme.equals(urlScheme) && this.zkClient != null) {
+ throw new IllegalStateException("Global, immutable 'urlScheme' already set for this node!");
+ }
+
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ public boolean useLiveNodesUrlScheme() {
+ return useLiveNodesUrlScheme;
+ }
+
+ public String getBaseUrlForNodeName(String nodeName) {
+ Objects.requireNonNull(nodeName,"node_name should not be null");
+ if (nodeName.indexOf('_') == -1) {
+ nodeName += '_'; // underscore required to indicate context
+ }
+ return Utils.getBaseUrlForNodeName(nodeName, getUrlSchemeForNodeName(nodeName));
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `UrlScheme.getBaseUrlForNodeName(...)` indirectly reads without synchronization from `this.liveNodes`. Potentially races with write in method `UrlScheme.onChange(...)`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private String urlScheme = HTTP;
+ private boolean useLiveNodesUrlScheme = false;
+ private SortedSet<String> liveNodes = null;
+ private SolrZkClient zkClient = null;
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public synchronized void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public synchronized void setUrlScheme(final String urlScheme) {
+ if (!this.urlScheme.equals(urlScheme) && this.zkClient != null) {
+ throw new IllegalStateException("Global, immutable 'urlScheme' already set for this node!");
+ }
+
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ public boolean useLiveNodesUrlScheme() {
+ return useLiveNodesUrlScheme;
+ }
+
+ public String getBaseUrlForNodeName(String nodeName) {
+ Objects.requireNonNull(nodeName,"node_name should not be null");
+ if (nodeName.indexOf('_') == -1) {
+ nodeName += '_'; // underscore required to indicate context
+ }
+ return Utils.getBaseUrlForNodeName(nodeName, getUrlSchemeForNodeName(nodeName));
+ }
+
+ public String getUrlSchemeForNodeName(final String nodeName) {
+ return useLiveNodesUrlScheme ? getSchemeFromLiveNode(nodeName).orElse(urlScheme) : urlScheme;
+ }
+
+ /**
+ * Given a URL with a replaceable parameter for the scheme, return a new URL with the correct scheme applied.
+ * @param url A URL to change the scheme (http|https)
+ * @return A new URL with the correct scheme
+ */
+ public String applyUrlScheme(final String url) {
+ Objects.requireNonNull(url, "URL must not be null!");
+
+ String updatedUrl;
+ if (useLiveNodesUrlScheme && liveNodes != null) {
+ updatedUrl = applyUrlSchemeFromLiveNodes(url);
+ if (updatedUrl != null) {
+ return updatedUrl;
+ }
+ }
+
+ // heal an incorrect scheme if needed, otherwise return null indicating no change
+ final int at = url.indexOf("://");
+ return (at == -1) ? (urlScheme + "://" + url) : urlScheme + url.substring(at);
+ }
+
+ public String getUrlScheme() {
+ return urlScheme;
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `UrlScheme.getUrlScheme()` reads without synchronization from `this.urlScheme`. Potentially races with write in method `UrlScheme.reset()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private String urlScheme = HTTP;
+ private boolean useLiveNodesUrlScheme = false;
+ private SortedSet<String> liveNodes = null;
+ private SolrZkClient zkClient = null;
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public synchronized void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public synchronized void setUrlScheme(final String urlScheme) {
+ if (!this.urlScheme.equals(urlScheme) && this.zkClient != null) {
+ throw new IllegalStateException("Global, immutable 'urlScheme' already set for this node!");
+ }
+
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ public boolean useLiveNodesUrlScheme() {
+ return useLiveNodesUrlScheme;
+ }
+
+ public String getBaseUrlForNodeName(String nodeName) {
+ Objects.requireNonNull(nodeName,"node_name should not be null");
+ if (nodeName.indexOf('_') == -1) {
+ nodeName += '_'; // underscore required to indicate context
+ }
+ return Utils.getBaseUrlForNodeName(nodeName, getUrlSchemeForNodeName(nodeName));
+ }
+
+ public String getUrlSchemeForNodeName(final String nodeName) {
+ return useLiveNodesUrlScheme ? getSchemeFromLiveNode(nodeName).orElse(urlScheme) : urlScheme;
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `UrlScheme.getUrlSchemeForNodeName(...)` indirectly reads without synchronization from `this.liveNodes`. Potentially races with write in method `UrlScheme.onChange(...)`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private String urlScheme = HTTP;
+ private boolean useLiveNodesUrlScheme = false;
+ private SortedSet<String> liveNodes = null;
+ private SolrZkClient zkClient = null;
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public synchronized void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `UrlScheme.isOnServer()` reads without synchronization from `this.zkClient`. Potentially races with write in method `UrlScheme.reset()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private String urlScheme = HTTP;
+ private boolean useLiveNodesUrlScheme = false;
+ private SortedSet<String> liveNodes = null;
+ private SolrZkClient zkClient = null;
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public synchronized void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public synchronized void setUrlScheme(final String urlScheme) {
+ if (!this.urlScheme.equals(urlScheme) && this.zkClient != null) {
+ throw new IllegalStateException("Global, immutable 'urlScheme' already set for this node!");
+ }
+
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ public boolean useLiveNodesUrlScheme() {
+ return useLiveNodesUrlScheme;
+ }
+
+ public String getBaseUrlForNodeName(String nodeName) {
+ Objects.requireNonNull(nodeName,"node_name should not be null");
+ if (nodeName.indexOf('_') == -1) {
+ nodeName += '_'; // underscore required to indicate context
+ }
+ return Utils.getBaseUrlForNodeName(nodeName, getUrlSchemeForNodeName(nodeName));
+ }
+
+ public String getUrlSchemeForNodeName(final String nodeName) {
+ return useLiveNodesUrlScheme ? getSchemeFromLiveNode(nodeName).orElse(urlScheme) : urlScheme;
+ }
+
+ /**
+ * Given a URL with a replaceable parameter for the scheme, return a new URL with the correct scheme applied.
+ * @param url A URL to change the scheme (http|https)
+ * @return A new URL with the correct scheme
+ */
+ public String applyUrlScheme(final String url) {
+ Objects.requireNonNull(url, "URL must not be null!");
+
+ String updatedUrl;
+ if (useLiveNodesUrlScheme && liveNodes != null) {
+ updatedUrl = applyUrlSchemeFromLiveNodes(url);
+ if (updatedUrl != null) {
+ return updatedUrl;
+ }
+ }
+
+ // heal an incorrect scheme if needed, otherwise return null indicating no change
+ final int at = url.indexOf("://");
+ return (at == -1) ? (urlScheme + "://" + url) : urlScheme + url.substring(at);
+ }
+
+ public String getUrlScheme() {
+ return urlScheme;
+ }
+
+ @Override
+ public synchronized boolean onChange(SortedSet<String> oldLiveNodes, SortedSet<String> newLiveNodes) {
+ if (useLiveNodesUrlScheme) {
+ liveNodes = newLiveNodes;
+ if (liveNodes != null) {
+ // we only need to clear the cached HTTP entries, keep the HTTPS
+ // as we don't really support a graceful downgrade from HTTPS -> HTTP
+ liveNodes.forEach(n -> {
+ if (HTTP.equals(nodeSchemeCache.get(n))) {
+ nodeSchemeCache.remove(n, HTTP);
+ }
+ });
+ } else {
+ nodeSchemeCache.clear();
+ }
+ } else {
+ nodeSchemeCache.clear();
+ liveNodes = null;
+ }
+ return !useLiveNodesUrlScheme;
+ }
+
+ private String applyUrlSchemeFromLiveNodes(final String url) {
+ String updatedUrl = null;
+ Optional<String> maybeFromLiveNode = getSchemeFromLiveNode(getNodeNameFromUrl(url));
+ if (maybeFromLiveNode.isPresent()) {
+ final int at = url.indexOf("://");
+ // replace the scheme on the url with the one from the matching live node entry
+ updatedUrl = maybeFromLiveNode.get() + ((at != -1) ? url.substring(at) : "://" + url);
+ }
+ return updatedUrl;
+ }
+
+ // Gets the urlScheme from the matching live node entry for this URL
+ private Optional<String> getSchemeFromLiveNode(final String nodeName) {
+ return (liveNodes != null && liveNodes.contains(nodeName)) ? Optional.ofNullable(getSchemeForLiveNode(nodeName)) : Optional.empty();
+ }
+
+ private String getNodeNameFromUrl(String url) {
+ final int at = url.indexOf("://");
+ if (at != -1) {
+ url = url.substring(at+3);
+ }
+ String hostAndPort = url;
+ String context = "";
+ int slashAt = url.indexOf('/');
+ if (slashAt != -1) {
+ hostAndPort = url.substring(0, slashAt);
+ // has context in url?s
+ if (slashAt < url.length()-1) {
+ context = url.substring(slashAt + 1);
+ }
+ }
+ if (!context.isEmpty()) {
+ context = URLEncoder.encode(trimLeadingAndTrailingSlashes(context), StandardCharsets.UTF_8);
+ }
+ return hostAndPort + "_" + context;
+ }
+
+ private String trimLeadingAndTrailingSlashes(final String in) {
+ String out = in;
+ if (out.startsWith("/")) {
+ out = out.substring(1);
+ }
+ if (out.endsWith("/")) {
+ out = out.substring(0, out.length() - 1);
+ }
+ return out;
+ }
+
+ @Override
+ public boolean onChange(Map<String, Object> properties) {
+ useLiveNodesUrlScheme = "true".equals(properties.getOrDefault(USE_LIVENODES_URL_SCHEME, "false"));
+ if (!useLiveNodesUrlScheme) {
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `UrlScheme.onChange(...)` reads without synchronization from `this.useLiveNodesUrlScheme`. Potentially races with write in method `UrlScheme.reset()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private String urlScheme = HTTP;
+ private boolean useLiveNodesUrlScheme = false;
+ private SortedSet<String> liveNodes = null;
+ private SolrZkClient zkClient = null;
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public synchronized void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public synchronized void setUrlScheme(final String urlScheme) {
+ if (!this.urlScheme.equals(urlScheme) && this.zkClient != null) {
+ throw new IllegalStateException("Global, immutable 'urlScheme' already set for this node!");
+ }
+
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ public boolean useLiveNodesUrlScheme() {
+ return useLiveNodesUrlScheme;
+ }
+
+ public String getBaseUrlForNodeName(String nodeName) {
+ Objects.requireNonNull(nodeName,"node_name should not be null");
+ if (nodeName.indexOf('_') == -1) {
+ nodeName += '_'; // underscore required to indicate context
+ }
+ return Utils.getBaseUrlForNodeName(nodeName, getUrlSchemeForNodeName(nodeName));
+ }
+
+ public String getUrlSchemeForNodeName(final String nodeName) {
+ return useLiveNodesUrlScheme ? getSchemeFromLiveNode(nodeName).orElse(urlScheme) : urlScheme;
+ }
+
+ /**
+ * Given a URL with a replaceable parameter for the scheme, return a new URL with the correct scheme applied.
+ * @param url A URL to change the scheme (http|https)
+ * @return A new URL with the correct scheme
+ */
+ public String applyUrlScheme(final String url) {
+ Objects.requireNonNull(url, "URL must not be null!");
+
+ String updatedUrl;
+ if (useLiveNodesUrlScheme && liveNodes != null) {
+ updatedUrl = applyUrlSchemeFromLiveNodes(url);
+ if (updatedUrl != null) {
+ return updatedUrl;
+ }
+ }
+
+ // heal an incorrect scheme if needed, otherwise return null indicating no change
+ final int at = url.indexOf("://");
+ return (at == -1) ? (urlScheme + "://" + url) : urlScheme + url.substring(at);
+ }
+
+ public String getUrlScheme() {
+ return urlScheme;
+ }
+
+ @Override
+ public synchronized boolean onChange(SortedSet<String> oldLiveNodes, SortedSet<String> newLiveNodes) {
+ if (useLiveNodesUrlScheme) {
+ liveNodes = newLiveNodes;
+ if (liveNodes != null) {
+ // we only need to clear the cached HTTP entries, keep the HTTPS
+ // as we don't really support a graceful downgrade from HTTPS -> HTTP
+ liveNodes.forEach(n -> {
+ if (HTTP.equals(nodeSchemeCache.get(n))) {
+ nodeSchemeCache.remove(n, HTTP);
+ }
+ });
+ } else {
+ nodeSchemeCache.clear();
+ }
+ } else {
+ nodeSchemeCache.clear();
+ liveNodes = null;
+ }
+ return !useLiveNodesUrlScheme;
+ }
+
+ private String applyUrlSchemeFromLiveNodes(final String url) {
+ String updatedUrl = null;
+ Optional<String> maybeFromLiveNode = getSchemeFromLiveNode(getNodeNameFromUrl(url));
+ if (maybeFromLiveNode.isPresent()) {
+ final int at = url.indexOf("://");
+ // replace the scheme on the url with the one from the matching live node entry
+ updatedUrl = maybeFromLiveNode.get() + ((at != -1) ? url.substring(at) : "://" + url);
+ }
+ return updatedUrl;
+ }
+
+ // Gets the urlScheme from the matching live node entry for this URL
+ private Optional<String> getSchemeFromLiveNode(final String nodeName) {
+ return (liveNodes != null && liveNodes.contains(nodeName)) ? Optional.ofNullable(getSchemeForLiveNode(nodeName)) : Optional.empty();
+ }
+
+ private String getNodeNameFromUrl(String url) {
+ final int at = url.indexOf("://");
+ if (at != -1) {
+ url = url.substring(at+3);
+ }
+ String hostAndPort = url;
+ String context = "";
+ int slashAt = url.indexOf('/');
+ if (slashAt != -1) {
+ hostAndPort = url.substring(0, slashAt);
+ // has context in url?s
+ if (slashAt < url.length()-1) {
+ context = url.substring(slashAt + 1);
+ }
+ }
+ if (!context.isEmpty()) {
+ context = URLEncoder.encode(trimLeadingAndTrailingSlashes(context), StandardCharsets.UTF_8);
+ }
+ return hostAndPort + "_" + context;
+ }
+
+ private String trimLeadingAndTrailingSlashes(final String in) {
+ String out = in;
+ if (out.startsWith("/")) {
+ out = out.substring(1);
+ }
+ if (out.endsWith("/")) {
+ out = out.substring(0, out.length() - 1);
+ }
+ return out;
+ }
+
+ @Override
+ public boolean onChange(Map<String, Object> properties) {
+ useLiveNodesUrlScheme = "true".equals(properties.getOrDefault(USE_LIVENODES_URL_SCHEME, "false"));
+ if (!useLiveNodesUrlScheme) {
+ nodeSchemeCache.clear();
+ liveNodes = null;
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `UrlScheme.onChange(...)` writes to field `this.liveNodes` outside of synchronization.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private String urlScheme = HTTP;
+ private boolean useLiveNodesUrlScheme = false;
+ private SortedSet<String> liveNodes = null;
+ private SolrZkClient zkClient = null;
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public synchronized void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public synchronized void setUrlScheme(final String urlScheme) {
+ if (!this.urlScheme.equals(urlScheme) && this.zkClient != null) {
+ throw new IllegalStateException("Global, immutable 'urlScheme' already set for this node!");
+ }
+
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ public boolean useLiveNodesUrlScheme() {
+ return useLiveNodesUrlScheme;
+ }
+
+ public String getBaseUrlForNodeName(String nodeName) {
+ Objects.requireNonNull(nodeName,"node_name should not be null");
+ if (nodeName.indexOf('_') == -1) {
+ nodeName += '_'; // underscore required to indicate context
+ }
+ return Utils.getBaseUrlForNodeName(nodeName, getUrlSchemeForNodeName(nodeName));
+ }
+
+ public String getUrlSchemeForNodeName(final String nodeName) {
+ return useLiveNodesUrlScheme ? getSchemeFromLiveNode(nodeName).orElse(urlScheme) : urlScheme;
+ }
+
+ /**
+ * Given a URL with a replaceable parameter for the scheme, return a new URL with the correct scheme applied.
+ * @param url A URL to change the scheme (http|https)
+ * @return A new URL with the correct scheme
+ */
+ public String applyUrlScheme(final String url) {
+ Objects.requireNonNull(url, "URL must not be null!");
+
+ String updatedUrl;
+ if (useLiveNodesUrlScheme && liveNodes != null) {
+ updatedUrl = applyUrlSchemeFromLiveNodes(url);
+ if (updatedUrl != null) {
+ return updatedUrl;
+ }
+ }
+
+ // heal an incorrect scheme if needed, otherwise return null indicating no change
+ final int at = url.indexOf("://");
+ return (at == -1) ? (urlScheme + "://" + url) : urlScheme + url.substring(at);
+ }
+
+ public String getUrlScheme() {
+ return urlScheme;
+ }
+
+ @Override
+ public synchronized boolean onChange(SortedSet<String> oldLiveNodes, SortedSet<String> newLiveNodes) {
+ if (useLiveNodesUrlScheme) {
+ liveNodes = newLiveNodes;
+ if (liveNodes != null) {
+ // we only need to clear the cached HTTP entries, keep the HTTPS
+ // as we don't really support a graceful downgrade from HTTPS -> HTTP
+ liveNodes.forEach(n -> {
+ if (HTTP.equals(nodeSchemeCache.get(n))) {
+ nodeSchemeCache.remove(n, HTTP);
+ }
+ });
+ } else {
+ nodeSchemeCache.clear();
+ }
+ } else {
+ nodeSchemeCache.clear();
+ liveNodes = null;
+ }
+ return !useLiveNodesUrlScheme;
+ }
+
+ private String applyUrlSchemeFromLiveNodes(final String url) {
+ String updatedUrl = null;
+ Optional<String> maybeFromLiveNode = getSchemeFromLiveNode(getNodeNameFromUrl(url));
+ if (maybeFromLiveNode.isPresent()) {
+ final int at = url.indexOf("://");
+ // replace the scheme on the url with the one from the matching live node entry
+ updatedUrl = maybeFromLiveNode.get() + ((at != -1) ? url.substring(at) : "://" + url);
+ }
+ return updatedUrl;
+ }
+
+ // Gets the urlScheme from the matching live node entry for this URL
+ private Optional<String> getSchemeFromLiveNode(final String nodeName) {
+ return (liveNodes != null && liveNodes.contains(nodeName)) ? Optional.ofNullable(getSchemeForLiveNode(nodeName)) : Optional.empty();
+ }
+
+ private String getNodeNameFromUrl(String url) {
+ final int at = url.indexOf("://");
+ if (at != -1) {
+ url = url.substring(at+3);
+ }
+ String hostAndPort = url;
+ String context = "";
+ int slashAt = url.indexOf('/');
+ if (slashAt != -1) {
+ hostAndPort = url.substring(0, slashAt);
+ // has context in url?s
+ if (slashAt < url.length()-1) {
+ context = url.substring(slashAt + 1);
+ }
+ }
+ if (!context.isEmpty()) {
+ context = URLEncoder.encode(trimLeadingAndTrailingSlashes(context), StandardCharsets.UTF_8);
+ }
+ return hostAndPort + "_" + context;
+ }
+
+ private String trimLeadingAndTrailingSlashes(final String in) {
+ String out = in;
+ if (out.startsWith("/")) {
+ out = out.substring(1);
+ }
+ if (out.endsWith("/")) {
+ out = out.substring(0, out.length() - 1);
+ }
+ return out;
+ }
+
+ @Override
+ public boolean onChange(Map<String, Object> properties) {
+ useLiveNodesUrlScheme = "true".equals(properties.getOrDefault(USE_LIVENODES_URL_SCHEME, "false"));
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `UrlScheme.onChange(...)` writes to field `this.useLiveNodesUrlScheme` outside of synchronization.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global vars in persisted state, such as the urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private String urlScheme = HTTP;
+ private boolean useLiveNodesUrlScheme = false;
+ private SortedSet<String> liveNodes = null;
+ private SolrZkClient zkClient = null;
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public synchronized void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
+ * the value on-the-fly.
+ * @param urlScheme The new URL scheme, either http or https.
+ */
+ public synchronized void setUrlScheme(final String urlScheme) {
+ if (!this.urlScheme.equals(urlScheme) && this.zkClient != null) {
+ throw new IllegalStateException("Global, immutable 'urlScheme' already set for this node!");
+ }
+
+ if (HTTP.equals(urlScheme) || HTTPS.equals(urlScheme)) {
+ this.urlScheme = urlScheme;
+ } else {
+ throw new IllegalArgumentException("Invalid urlScheme: "+urlScheme);
+ }
+ }
+
+ public boolean useLiveNodesUrlScheme() {
+ return useLiveNodesUrlScheme;
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `UrlScheme.useLiveNodesUrlScheme()` reads without synchronization from `this.useLiveNodesUrlScheme`. Potentially races with write in method `UrlScheme.reset()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] muse-dev[bot] commented on a change in pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
muse-dev[bot] commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r517712472
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/Replica.java
##########
@@ -235,11 +234,11 @@ public String getName() {
}
public String getCoreUrl() {
- return ZkCoreNodeProps.getCoreUrl(getStr(ZkStateReader.BASE_URL_PROP), core);
+ return ZkCoreNodeProps.getCoreUrl(getBaseUrl(), core);
Review comment:
*NULL_DEREFERENCE:* object returned by `getBaseUrl()` could be null and is dereferenced by call to `getCoreUrl(...)` at line 237.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-729783538
Hi @noblepaul thanks for taking a look ... so we decided to not try to handle the upgrade to TLS using a rolling restart (as described in my comment ^) in this PR ... also see Anshum's comments. I initially had a way to get the "current" urlScheme for each live node as with a rolling restart, you'll have a mix of nodes with TLS enabled and some not yet, but we felt that could be a little trappy b/c it really doesn't address the client applications. So our advice now is just suck it up and take the down time to enable TLS. Basically, we don't want to promise the community a zero-downtime upgrade to enable TLS, because it is a hard thing to promise. The live nodes approach (see commit history in this PR) works on the server side, but doesn't address client applications. Probably other weird issues too ... FWIW ~ that's the current experience as well, so we're not any worse off. I find it highly unlikely that users will enable TLS after building up a large production cluster anyway,
that seems like it wouldn't happen in practice.
Regarding migrating to this: I don't think these changes would require any migration process. Currently, `node_name` is stored in the state in ZK (as you know), so the stored `base_url` will just be ignored and re-created when reading from ZK using the `node_name`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-997095585
@broberson721 ~ I'm doing the work for this issue here: https://github.com/apache/solr/pull/460 ... if you want to follow along. Thanks again for reporting, obviously I hadn't considered the use-case of multiple clients in the same JVM talking to different Solr clusters ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-996264243
yeah we need to fix that ... the singleton works great for the server-side but doesn't work for multiple clusters with different SSL config as you describe ... I'll come up with something for 8.11.2 and 9. In the meantime, I guess you can use a pre-8.8 SolrJ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-713631088
Switched this to draft as we need a better solution for migrating from using `http` -> `https` using a rolling restart where some nodes will still be using `http` and others `https` (temporarily). Currently, switching the urlScheme over to https globally before all nodes have restarted could lead to unnecessary thrashing and cause downtime as leader elections would be effected while the rolling restart is progressing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] muse-dev[bot] commented on a change in pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
muse-dev[bot] commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r509829039
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -448,6 +447,11 @@ public boolean isClosed() {
return cc.isShutDown();
}});
+ // setup the scheme before updating cluster state
+ setGlobalUrlSchemeFromClusterProps(zkClient);
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `ZkController(...)` indirectly writes to field `noggit.JSONParser.devNull.buf` outside of synchronization.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] muse-dev[bot] commented on a change in pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
muse-dev[bot] commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r508905260
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java
##########
@@ -798,7 +799,11 @@ public boolean isClosed() {
}
public String getLeaderUrl(String collection, String shard, int timeout) throws InterruptedException {
- ZkCoreNodeProps props = new ZkCoreNodeProps(getLeaderRetry(collection, shard, timeout));
+ Replica replica = getLeaderRetry(collection, shard, timeout);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ZkStateReader.getLeaderUrl(...)` indirectly reads without synchronization from `cloud.UrlScheme.INSTANCE.urlScheme`. Potentially races with write in method `ZkStateReader.forciblyRefreshAllClusterStateSlow()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] murblanc commented on pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
murblanc commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-713418066
Why store `${scheme}://` in Zookeeper? We could store the rest of the URL (`host:port/path`) given we'll rebuild the URL from configuration available elsewhere.
For existing URLs with a scheme we could just remove/ignore the scheme and use the configured one instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
noblepaul commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-729684303
Before I go further, can I know
* Does this work for a rolling restart?
* How does one migrate to this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] noblepaul edited a comment on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
noblepaul edited a comment on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-731661638
> we don't want to promise the community a zero-downtime upgrade to enable TLS, because it is a hard thing to promise
I don't have a problem if there is a downtime when you upgrade from one schem to another. I only worry if the scheme remains same and a rolling restart is not possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on a change in pull request #2010: SOLR-12182: Store URL scheme as replaceable parameter in state.json
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r508868917
##########
File path: solr/core/src/java/org/apache/solr/core/ZkContainer.java
##########
@@ -263,4 +275,27 @@ public void close() {
public ExecutorService getCoreZkRegisterExecutorService() {
return coreZkRegister;
}
+
+ private void setGlobalUrlScheme(final ZkController zkCtrl) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String urlScheme = zkCtrl.getZkStateReader().getClusterProperty(ZkStateReader.URL_SCHEME, null);
+ if (StringUtils.isNotEmpty(urlScheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ GlobalStateVars.singleton().setUrlScheme(urlScheme);
+ } else {
+ final String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ // it's OK to set the cluster prop to https b/c we're making the cluster more secure, not less
+ // wouldn't do this for http though
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. Updating the cluster property to match.");
Review comment:
I'll just remove this, not strictly necessary (was trying to save the user some work but could have side-effects)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
noblepaul commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-731661859
LGTM
👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] broberson721 commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
broberson721 commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-996257742
UrlScheme.INSTANCE became a singleton with this PR. That is causing failures for use cases where a single client connects to some clusters with SSL enabled and some without SSL enabled.. Are there any plans to fix this in a future release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
thelabdude commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-735976873
thank you for the review @noblepaul and @anshumg ... going to merge to master now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] anshumg commented on a change in pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
anshumg commented on a change in pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#discussion_r520351889
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ *
+ * Client applications should not use this class directly; it is only included in SolrJ because Replica
+ * and ZkNodeProps depend on it.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private volatile String urlScheme = HTTP;
+ private volatile boolean useLiveNodesUrlScheme = false;
+ private volatile SortedSet<String> liveNodes = null;
+ private volatile SolrZkClient zkClient = null;
+
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
Review comment:
Should we instead mention that Solr doesn't support partial TLS enabled clusters?
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/UrlScheme.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.common.cloud;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.Objects;
+import java.util.SortedSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.cloud.ZkStateReader.URL_SCHEME;
+
+/**
+ * Singleton access to global urlScheme, which although is stored in ZK as a cluster property
+ * really should be treated like a static global that is set at initialization and not altered after.
+ *
+ * Client applications should not use this class directly; it is only included in SolrJ because Replica
+ * and ZkNodeProps depend on it.
+ */
+public enum UrlScheme implements LiveNodesListener, ClusterPropertiesListener {
+ INSTANCE;
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String HTTP = "http";
+ public static final String HTTPS = "https";
+ public static final String HTTPS_PORT_PROP = "solr.jetty.https.port";
+ public static final String USE_LIVENODES_URL_SCHEME = "ext.useLiveNodesUrlScheme";
+
+ private volatile String urlScheme = HTTP;
+ private volatile boolean useLiveNodesUrlScheme = false;
+ private volatile SortedSet<String> liveNodes = null;
+ private volatile SolrZkClient zkClient = null;
+
+ private final ConcurrentMap<String,String> nodeSchemeCache = new ConcurrentHashMap<>();
+
+ /**
+ * Called during ZkController initialization to set the urlScheme based on cluster properties.
+ * @param client The SolrZkClient needed to read cluster properties from ZK.
+ * @throws IOException If a connection or other I/O related error occurs while reading from ZK.
+ */
+ public void initFromClusterProps(final SolrZkClient client) throws IOException {
+ this.zkClient = client;
+
+ // Have to go directly to the cluster props b/c this needs to happen before ZkStateReader does its thing
+ ClusterProperties clusterProps = new ClusterProperties(client);
+ this.useLiveNodesUrlScheme =
+ "true".equals(clusterProps.getClusterProperty(UrlScheme.USE_LIVENODES_URL_SCHEME, "false"));
+ setUrlSchemeFromClusterProps(clusterProps.getClusterProperties());
+ }
+
+ private void setUrlSchemeFromClusterProps(Map<String,Object> props) {
+ // Set the global urlScheme from cluster prop or if that is not set, look at the urlScheme sys prop
+ final String scheme = (String)props.get(ZkStateReader.URL_SCHEME);
+ if (StringUtils.isNotEmpty(scheme)) {
+ // track the urlScheme in a global so we can use it during ZK read / write operations for cluster state objects
+ this.urlScheme = HTTPS.equals(scheme) ? HTTPS : HTTP;
+ } else {
+ String urlSchemeFromSysProp = System.getProperty(URL_SCHEME, HTTP);
+ if (HTTPS.equals(urlSchemeFromSysProp)) {
+ log.warn("Cluster property 'urlScheme' not set but system property is set to 'https'. " +
+ "You should set the cluster property and restart all nodes for consistency.");
+ }
+ // TODO: We may not want this? See: https://issues.apache.org/jira/browse/SOLR-10202
+ this.urlScheme = HTTPS.equals(urlSchemeFromSysProp) ? HTTPS : HTTP;
+ }
+ }
+
+ public boolean isOnServer() {
+ return zkClient != null;
+ }
+
+ /**
+ * Set the global urlScheme variable; ideally this should be immutable once set, but some tests rely on changing
Review comment:
👍🏽
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] anshumg commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
anshumg commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-725022361
We should restrict this PR to just handle the removal of persisted base_url in ZK.
For users who intend to enable/disable TLS for their cluster, they should plan for downtime and as guaranteeing a safe rolling restart during such a transition is not possible right now.
I think we're better off not trying for a best effort during a rolling restart here and removing the live_nodes part of the code change to keep things simple.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] broberson721 commented on pull request #2010: SOLR-12182: Don't persist base_url in ZK as the scheme is variable, compute from node_name instead
Posted by GitBox <gi...@apache.org>.
broberson721 commented on pull request #2010:
URL: https://github.com/apache/lucene-solr/pull/2010#issuecomment-996278823
Understood. And thanks. We're hoping a fix gets out there before we have to patch due to security issues. I think the big security issue with 8.7 (CVE-2021-29943) doesn't affect us. But it's usually just a matter of time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org