You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2021/12/17 22:55:41 UTC
[GitHub] [solr] thelabdude opened a new pull request #460: SOLR-15587: Don't use the UrlScheme singleton on the client-side
thelabdude opened a new pull request #460:
URL: https://github.com/apache/solr/pull/460
https://issues.apache.org/jira/browse/SOLR-15587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on a change in pull request #460: SOLR-15587: Don't use the UrlScheme singleton on the client-side
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #460:
URL: https://github.com/apache/solr/pull/460#discussion_r799910426
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/SplitShardCmd.java
##########
@@ -469,6 +469,7 @@ public boolean split(ClusterState clusterState, ZkNodeProps message, NamedList<O
ZkStateReader.REPLICA_TYPE, replicaPosition.type.name(),
ZkStateReader.STATE_PROP, Replica.State.DOWN.toString(),
ZkStateReader.NODE_NAME_PROP, subShardNodeName,
+ ZkStateReader.BASE_URL_PROP, zkStateReader.getBaseUrlForNodeName(subShardNodeName),
Review comment:
Inspired by Mike ~ there's no more UrlScheme!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on a change in pull request #460: SOLR-15587: Don't use the UrlScheme singleton on the client-side
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #460:
URL: https://github.com/apache/solr/pull/460#discussion_r771744339
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java
##########
@@ -976,9 +975,6 @@ private void loadClusterProperties() {
this.clusterProperties = ClusterProperties.convertCollectionDefaultsToNestedFormat(properties);
log.debug("Loaded cluster properties: {}", this.clusterProperties);
- // Make the urlScheme globally accessible
- UrlScheme.INSTANCE.setUrlScheme(getClusterProperty(ZkStateReader.URL_SCHEME, HTTP));
Review comment:
We don't want to set the Singleton `urlScheme` in the `ZkStateReader` since that runs on the client side too, which would force all clients to use the same scheme if multiple are talking to different clusters (or it might flip-flop around too, which would be really bad). It's best for the "client" side to get back whatever is stored from the server, which is why we have to store `base_url` in the replica state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on a change in pull request #460: SOLR-15587: Don't use the UrlScheme singleton on the client-side
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #460:
URL: https://github.com/apache/solr/pull/460#discussion_r786990164
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/SplitShardCmd.java
##########
@@ -469,6 +469,7 @@ public boolean split(ClusterState clusterState, ZkNodeProps message, NamedList<O
ZkStateReader.REPLICA_TYPE, replicaPosition.type.name(),
ZkStateReader.STATE_PROP, Replica.State.DOWN.toString(),
ZkStateReader.NODE_NAME_PROP, subShardNodeName,
+ ZkStateReader.BASE_URL_PROP, zkStateReader.getBaseUrlForNodeName(subShardNodeName),
Review comment:
Good point! All the code in `core` should use `UrlScheme.INSTANCE` and all the code in `solrj` should use the `ZkStateReader#getBaseUrlForNodeName`. Will update across the codebase to make this consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] madrob commented on a change in pull request #460: SOLR-15587: Don't use the UrlScheme singleton on the client-side
Posted by GitBox <gi...@apache.org>.
madrob commented on a change in pull request #460:
URL: https://github.com/apache/solr/pull/460#discussion_r786969036
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/SplitShardCmd.java
##########
@@ -469,6 +469,7 @@ public boolean split(ClusterState clusterState, ZkNodeProps message, NamedList<O
ZkStateReader.REPLICA_TYPE, replicaPosition.type.name(),
ZkStateReader.STATE_PROP, Replica.State.DOWN.toString(),
ZkStateReader.NODE_NAME_PROP, subShardNodeName,
+ ZkStateReader.BASE_URL_PROP, zkStateReader.getBaseUrlForNodeName(subShardNodeName),
Review comment:
I'm confused why this isn't UrlScheme.INSTANCE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude merged pull request #460: SOLR-15587: Don't use the UrlScheme singleton on the client-side
Posted by GitBox <gi...@apache.org>.
thelabdude merged pull request #460:
URL: https://github.com/apache/solr/pull/460
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on a change in pull request #460: SOLR-15587: Don't use the UrlScheme singleton on the client-side
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #460:
URL: https://github.com/apache/solr/pull/460#discussion_r779879892
##########
File path: solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java
##########
@@ -976,9 +975,6 @@ private void loadClusterProperties() {
this.clusterProperties = ClusterProperties.convertCollectionDefaultsToNestedFormat(properties);
log.debug("Loaded cluster properties: {}", this.clusterProperties);
- // Make the urlScheme globally accessible
- UrlScheme.INSTANCE.setUrlScheme(getClusterProperty(ZkStateReader.URL_SCHEME, HTTP));
Review comment:
This has now been found in the wild, see: https://issues.apache.org/jira/browse/SOLR-15364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] thelabdude commented on a change in pull request #460: SOLR-15587: Don't use the UrlScheme singleton on the client-side
Posted by GitBox <gi...@apache.org>.
thelabdude commented on a change in pull request #460:
URL: https://github.com/apache/solr/pull/460#discussion_r779683183
##########
File path: solr/core/src/java/org/apache/solr/util/UrlScheme.java
##########
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.solr.common.cloud;
+package org.apache.solr.util;
Review comment:
In the backport to 8.11.x, I'll keep `UrlScheme` in Solrj in the `org.apache.solr.common.cloud` package as client apps may be using it. It will be marked as Deprecated and removed from `solrj` in 9.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] anshumg commented on a change in pull request #460: SOLR-15587: Don't use the UrlScheme singleton on the client-side
Posted by GitBox <gi...@apache.org>.
anshumg commented on a change in pull request #460:
URL: https://github.com/apache/solr/pull/460#discussion_r787467328
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CollApiCmds.java
##########
@@ -33,7 +33,7 @@
import org.apache.solr.common.cloud.ClusterState;
import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.SolrZkClient;
-import org.apache.solr.common.cloud.UrlScheme;
+import org.apache.solr.util.UrlScheme;
Review comment:
Might want to fix the import order here.
##########
File path: solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpClusterStateSSLTest.java
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.client.solrj.impl;
+
+import java.net.URL;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.cloud.SolrCloudTestCase;
+import org.apache.solr.common.cloud.ClusterState;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.util.RandomizeSSL;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+@RandomizeSSL(1.0)
+public class HttpClusterStateSSLTest extends SolrCloudTestCase {
+
+ @BeforeClass
+ public static void setupClusterWithSSL() throws Exception {
+ System.setProperty("solr.storeBaseUrl", "true");
+ configureCluster(1)
+ .addConfig("conf", getFile("solrj").toPath().resolve("solr").resolve("configsets").resolve("streaming").resolve("conf"))
+ .configure();
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testHttpClusterStateWithSSL() throws Exception {
+ URL url0 = cluster.getJettySolrRunner(0).getBaseUrl();
+ assertEquals("https", url0.getProtocol());
+
+ final String collectionId = "test-https-cluster-state";
+ final int numShards = 3;
+ final int rf = 2;
+ final int expectedReplicas = numShards * rf;
+ CollectionAdminRequest.createCollection(collectionId, "conf", numShards, rf)
+ .setPerReplicaState(false)
+ .process(cluster.getSolrClient());
+ cluster.waitForActiveCollection(collectionId, numShards, numShards * rf);
+
+ // verify the base_url is actually stored with https in it on the server-side
+ byte[] stateJsonBytes = cluster.getZkClient().getData(ZkStateReader.getCollectionPath(collectionId), null, null, true);
+ assertNotNull(stateJsonBytes);
+ Map<String, Object> replicasMap =
+ (Map<String, Object>) Utils.getObjectByPath(Utils.fromJSON(stateJsonBytes), false, "/" + collectionId + "/shards/shard1/replicas");
+ assertEquals(2, replicasMap.size());
+ for (Object replicaObj : replicasMap.values()) {
+ Map<String, Object> replicaData = (Map<String, Object>) replicaObj;
+ String baseUrl = (String) replicaData.get("base_url");
+ assertTrue(baseUrl != null && baseUrl.startsWith("https://"));
+ }
+
+ // verify the http derived cluster state (on the client side) agrees with what the server stored
+ try (CloudSolrClient httpBasedCloudSolrClient = new CloudSolrClient.Builder(Collections.singletonList(url0.toExternalForm())).build()) {
+ ClusterStateProvider csp = httpBasedCloudSolrClient.getClusterStateProvider();
+ assertTrue(csp instanceof HttpClusterStateProvider);
+ verifyUrlSchemeInClusterState(csp.getClusterState(), collectionId, expectedReplicas);
+ }
+
+ // http2
+ try (CloudHttp2SolrClient http2BasedClient = new CloudHttp2SolrClient.Builder(Collections.singletonList(url0.toExternalForm())).build()) {
+ ClusterStateProvider csp = http2BasedClient.getClusterStateProvider();
+ assertTrue(csp instanceof Http2ClusterStateProvider);
+ verifyUrlSchemeInClusterState(csp.getClusterState(), collectionId, expectedReplicas);
+ }
+
+ // Zk cluster state now
+ ClusterStateProvider csp = cluster.getSolrClient().getClusterStateProvider();
+ assertTrue(csp instanceof ZkClientClusterStateProvider);
+ verifyUrlSchemeInClusterState(csp.getClusterState(), collectionId, expectedReplicas);
+ }
+
+ private void verifyUrlSchemeInClusterState(final ClusterState cs, final String collectionId, final int expectedReplicas) {
Review comment:
The `collectionId` is always `test-https-cluster-state`. Do you want to just remove this parameter and declare it as static?
##########
File path: solr/solrj/src/test/org/apache/solr/client/solrj/impl/HttpClusterStateSSLTest.java
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.client.solrj.impl;
+
+import java.net.URL;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.cloud.SolrCloudTestCase;
+import org.apache.solr.common.cloud.ClusterState;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.util.RandomizeSSL;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+@RandomizeSSL(1.0)
+public class HttpClusterStateSSLTest extends SolrCloudTestCase {
+
+ @BeforeClass
+ public static void setupClusterWithSSL() throws Exception {
+ System.setProperty("solr.storeBaseUrl", "true");
+ configureCluster(1)
+ .addConfig("conf", getFile("solrj").toPath().resolve("solr").resolve("configsets").resolve("streaming").resolve("conf"))
+ .configure();
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testHttpClusterStateWithSSL() throws Exception {
+ URL url0 = cluster.getJettySolrRunner(0).getBaseUrl();
+ assertEquals("https", url0.getProtocol());
+
+ final String collectionId = "test-https-cluster-state";
+ final int numShards = 3;
+ final int rf = 2;
+ final int expectedReplicas = numShards * rf;
+ CollectionAdminRequest.createCollection(collectionId, "conf", numShards, rf)
+ .setPerReplicaState(false)
+ .process(cluster.getSolrClient());
+ cluster.waitForActiveCollection(collectionId, numShards, numShards * rf);
+
+ // verify the base_url is actually stored with https in it on the server-side
+ byte[] stateJsonBytes = cluster.getZkClient().getData(ZkStateReader.getCollectionPath(collectionId), null, null, true);
+ assertNotNull(stateJsonBytes);
+ Map<String, Object> replicasMap =
+ (Map<String, Object>) Utils.getObjectByPath(Utils.fromJSON(stateJsonBytes), false, "/" + collectionId + "/shards/shard1/replicas");
+ assertEquals(2, replicasMap.size());
+ for (Object replicaObj : replicasMap.values()) {
+ Map<String, Object> replicaData = (Map<String, Object>) replicaObj;
+ String baseUrl = (String) replicaData.get("base_url");
+ assertTrue(baseUrl != null && baseUrl.startsWith("https://"));
+ }
+
+ // verify the http derived cluster state (on the client side) agrees with what the server stored
+ try (CloudSolrClient httpBasedCloudSolrClient = new CloudSolrClient.Builder(Collections.singletonList(url0.toExternalForm())).build()) {
+ ClusterStateProvider csp = httpBasedCloudSolrClient.getClusterStateProvider();
+ assertTrue(csp instanceof HttpClusterStateProvider);
+ verifyUrlSchemeInClusterState(csp.getClusterState(), collectionId, expectedReplicas);
+ }
+
+ // http2
+ try (CloudHttp2SolrClient http2BasedClient = new CloudHttp2SolrClient.Builder(Collections.singletonList(url0.toExternalForm())).build()) {
+ ClusterStateProvider csp = http2BasedClient.getClusterStateProvider();
+ assertTrue(csp instanceof Http2ClusterStateProvider);
+ verifyUrlSchemeInClusterState(csp.getClusterState(), collectionId, expectedReplicas);
+ }
+
+ // Zk cluster state now
+ ClusterStateProvider csp = cluster.getSolrClient().getClusterStateProvider();
+ assertTrue(csp instanceof ZkClientClusterStateProvider);
+ verifyUrlSchemeInClusterState(csp.getClusterState(), collectionId, expectedReplicas);
+ }
+
+ private void verifyUrlSchemeInClusterState(final ClusterState cs, final String collectionId, final int expectedReplicas) {
Review comment:
Same with `expectedReplicas` (6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org