You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "jdyer1 (via GitHub)" <gi...@apache.org> on 2024/10/31 15:03:49 UTC
[PR] SOLR-17516: LBHttpSolrClient: support HttpJdkSolrClient (Generic Version) [solr]
jdyer1 opened a new pull request, #2828:
URL: https://github.com/apache/solr/pull/2828
Similar to #2805, this is a minimal PR to add Load Balancing functionality for `HttpJdkSolrClient`. Included is test refactoring, adding the *JdkClient as a random choice for the integration test.
Whereas #2805 created a class hierarchy, this makes `LBHttp2SolrClient` generic. Adding a generic parameter to this class will not break backwards-compatibility. This seems a better option that #2805.
One downside is we retain the name `LBHttp2SolrClient` although it is no longer strictly to be used with `Http2SolrClient`. However, the legacy implementation already uses the preferred name: `LBHttpSolrClient`. Whenever the legacy implementation is removed, I recommend also renaming `LBHttp2SolrClient` to `LBHttpSolrClient`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17516: LBHttpSolrClient: support HttpJdkSolrClient (Generic Version) [solr]
Posted by "jdyer1 (via GitHub)" <gi...@apache.org>.
jdyer1 commented on code in PR #2828:
URL: https://github.com/apache/solr/pull/2828#discussion_r1825908232
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpJdkSolrClient.java:
##########
@@ -173,8 +173,8 @@ public NamedList<Object> request(SolrRequest<?> solrRequest, String collection)
"Timeout occurred while waiting response from server at: " + pReq.url, e);
} catch (SolrException se) {
throw se;
- } catch (RuntimeException re) {
- throw new SolrServerException(re);
+ } catch (IOException | RuntimeException e) {
Review Comment:
I found that `LBHttpSolrClient` needs a `SolrServerException` to detect a down node. Yet we still can throw an `IOException` here during `prepareRequest`. This roughly mimics the exception handling in `Http2SolrClient`.
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttp2SolrClient.java:
##########
@@ -37,9 +37,9 @@
import org.slf4j.MDC;
/**
- * LBHttp2SolrClient or "LoadBalanced LBHttp2SolrClient" is a load balancing wrapper around {@link
- * Http2SolrClient}. This is useful when you have multiple Solr endpoints and requests need to be
- * Load Balanced among them.
+ * LBHttp2SolrClient or "LoadBalanced Http Solr Client" is a load balancing wrapper around a Http
Review Comment:
I improved the wording.
##########
solr/solrj/src/test/org/apache/solr/client/solrj/impl/LBHttp2SolrClientTest.java:
##########
@@ -50,8 +52,8 @@ public void testLBHttp2SolrClientWithTheseParamNamesInTheUrl() {
try (Http2SolrClient http2SolrClient =
new Http2SolrClient.Builder(url).withTheseParamNamesInTheUrl(urlParamNames).build();
Review Comment:
Actually, 4/5 test methods here are using a mock client. The `withTheseParamNamesInTheUrl` case is being tested with the clients' own unit tests. I did add a random choice to `LBHttp2SolrClientIntegrationTest` (renamed). I think this is pretty good coverage already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17516: LBHttpSolrClient: support HttpJdkSolrClient (Generic Version) [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #2828:
URL: https://github.com/apache/solr/pull/2828#discussion_r1844643643
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttp2SolrClient.java:
##########
@@ -210,18 +210,23 @@ private CompletableFuture<NamedList<Object>> doAsyncRequest(
RetryListener listener) {
String baseUrl = endpoint.toString();
rsp.server = baseUrl;
- req.getRequest().setBasePath(baseUrl);
- CompletableFuture<NamedList<Object>> future =
- ((HttpSolrClientBase) getClient(endpoint)).requestAsync(req.getRequest());
- future.whenComplete(
- (result, throwable) -> {
- if (!future.isCompletedExceptionally()) {
- onSuccessfulRequest(result, endpoint, rsp, isZombie, listener);
- } else if (!future.isCancelled()) {
- onFailedRequest(throwable, endpoint, isNonRetryable, isZombie, listener);
- }
- });
- return future;
+ final var client = (Http2SolrClient) getClient(endpoint);
+ try {
+ CompletableFuture<NamedList<Object>> future =
+ client.requestWithBaseUrl(baseUrl, (c) -> c.requestAsync(req.getRequest()));
+ future.whenComplete(
+ (result, throwable) -> {
+ if (!future.isCompletedExceptionally()) {
+ onSuccessfulRequest(result, endpoint, rsp, isZombie, listener);
+ } else if (!future.isCancelled()) {
+ onFailedRequest(throwable, endpoint, isNonRetryable, isZombie, listener);
+ }
+ });
+ return future;
+ } catch (SolrServerException | IOException e) {
+ // Unreachable, since 'requestWithBaseUrl' above is running the request asynchronously
Review Comment:
an "assert false" would be helpful.
I suppose a requestAsyncWithBaseUrl would be nice but not a big deal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17516: LBHttpSolrClient: support HttpJdkSolrClient (Generic Version) [solr]
Posted by "jdyer1 (via GitHub)" <gi...@apache.org>.
jdyer1 commented on code in PR #2828:
URL: https://github.com/apache/solr/pull/2828#discussion_r1827873836
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpJdkSolrClient.java:
##########
@@ -173,8 +173,8 @@ public NamedList<Object> request(SolrRequest<?> solrRequest, String collection)
"Timeout occurred while waiting response from server at: " + pReq.url, e);
} catch (SolrException se) {
throw se;
- } catch (RuntimeException re) {
- throw new SolrServerException(re);
+ } catch (IOException | RuntimeException e) {
Review Comment:
Indeed, the doc-comment on the `LBSolrClient` request method says,
> If a request fails due to an IOException, the server is moved to the dead pool
So I restored the previous behavior in `HttpJdkSolrClient` and made the modification in `LBSolrClient`. This new behavior is covered by `LBHttp2SolrClientIntegrationTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17516: LBHttpSolrClient: support HttpJdkSolrClient (Generic Version) [solr]
Posted by "jdyer1 (via GitHub)" <gi...@apache.org>.
jdyer1 commented on PR #2828:
URL: https://github.com/apache/solr/pull/2828#issuecomment-2479258656
@gerlowskija @dsmiley This PR required significant changes to merge with the incoming changes from SOLR-17256. I would appreciate another round of review before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17516: LBHttpSolrClient: support HttpJdkSolrClient (Generic Version) [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #2828:
URL: https://github.com/apache/solr/pull/2828#discussion_r1827982607
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBSolrClient.java:
##########
@@ -699,11 +699,20 @@ public NamedList<Object> request(
if (e.getRootCause() instanceof IOException) {
ex = e;
moveAliveToDead(wrapper);
- if (justFailed == null) justFailed = new HashMap<>();
+ if (justFailed == null) {
+ justFailed = new HashMap<>();
+ }
Review Comment:
It's a shame to see duplication in getRootCause IOException detection with the dedicated clause. Maybe it'd be cleaner to have another try-catch just around calling request() above that detects the IOException and throws it? Just a thought; I leave it to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17516: LBHttpSolrClient: support HttpJdkSolrClient (Generic Version) [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #2828:
URL: https://github.com/apache/solr/pull/2828#discussion_r1826120759
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpJdkSolrClient.java:
##########
@@ -173,8 +173,8 @@ public NamedList<Object> request(SolrRequest<?> solrRequest, String collection)
"Timeout occurred while waiting response from server at: " + pReq.url, e);
} catch (SolrException se) {
throw se;
- } catch (RuntimeException re) {
- throw new SolrServerException(re);
+ } catch (IOException | RuntimeException e) {
Review Comment:
Actually, maybe LBHttpSolrClient should be enhanced to catch the IOException to behave properly here? Otherwise, it seems wrong just looking at this SolrClient to throw/not-throw based on some wrapping/delegating client unless we spell out these things in the documented semantics of this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17516: LBHttpSolrClient: support HttpJdkSolrClient (Generic Version) [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #2828:
URL: https://github.com/apache/solr/pull/2828#discussion_r1825196815
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpJdkSolrClient.java:
##########
@@ -173,8 +173,8 @@ public NamedList<Object> request(SolrRequest<?> solrRequest, String collection)
"Timeout occurred while waiting response from server at: " + pReq.url, e);
} catch (SolrException se) {
throw se;
- } catch (RuntimeException re) {
- throw new SolrServerException(re);
+ } catch (IOException | RuntimeException e) {
Review Comment:
The request method declares it throws IOException; shouldn't we propagate it instead of wrap?
##########
solr/solrj/src/test/org/apache/solr/client/solrj/impl/LBHttp2SolrClientTest.java:
##########
@@ -50,8 +52,8 @@ public void testLBHttp2SolrClientWithTheseParamNamesInTheUrl() {
try (Http2SolrClient http2SolrClient =
new Http2SolrClient.Builder(url).withTheseParamNamesInTheUrl(urlParamNames).build();
Review Comment:
perhaps some tests methods shouldn't care in theory what the httpSolrClient impl is exactly but it'd be nice to get one randomly or via parameterization even better. Not sure if that'd be too difficult.
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttp2SolrClient.java:
##########
@@ -37,9 +37,9 @@
import org.slf4j.MDC;
/**
- * LBHttp2SolrClient or "LoadBalanced LBHttp2SolrClient" is a load balancing wrapper around {@link
- * Http2SolrClient}. This is useful when you have multiple Solr endpoints and requests need to be
- * Load Balanced among them.
+ * LBHttp2SolrClient or "LoadBalanced Http Solr Client" is a load balancing wrapper around a Http
Review Comment:
Don't need to share the name of our own class redundantly
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttp2SolrClient.java:
##########
@@ -284,16 +286,16 @@ private void onFailedRequest(
}
}
- public static class Builder {
+ public static class Builder<C extends HttpSolrClientBase> {
- private final Http2SolrClient http2SolrClient;
- private final Endpoint[] solrEndpoints;
+ private final C solrClient;
+ private final LBSolrClient.Endpoint[] solrEndpoints;
private long aliveCheckIntervalMillis =
TimeUnit.MILLISECONDS.convert(60, TimeUnit.SECONDS); // 1 minute between checks
protected String defaultCollection;
- public Builder(Http2SolrClient http2Client, Endpoint... endpoints) {
- this.http2SolrClient = http2Client;
+ public Builder(C http2Client, LBSolrClient.Endpoint... endpoints) {
Review Comment:
Can you add a static import for that Endpoint inner class so it looks simple as it was?
minor but can you please rename "http2Client" to incorporate the name "Solr" like just call it "solrClient" is fine. I like a SolrClient and HttpClient to not be confusable.
##########
solr/solrj/src/test/org/apache/solr/client/solrj/impl/MockTrustManager.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.impl;
+
+import java.net.Socket;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLEngine;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509ExtendedTrustManager;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.util.SSLTestConfig;
+
+/**
+ * Taken from: https://www.baeldung.com/java-httpclient-ssl sec 4.1, 2024/02/12. This is an
+ * all-trusting Trust Manager. Works with self-signed certificates.
+ */
+public class MockTrustManager extends X509ExtendedTrustManager {
Review Comment:
nice to see this separated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17516: LBHttpSolrClient: support HttpJdkSolrClient (Generic Version) [solr]
Posted by "jdyer1 (via GitHub)" <gi...@apache.org>.
jdyer1 merged PR #2828:
URL: https://github.com/apache/solr/pull/2828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17516: LBHttpSolrClient: support HttpJdkSolrClient (Generic Version) [solr]
Posted by "jdyer1 (via GitHub)" <gi...@apache.org>.
jdyer1 commented on code in PR #2828:
URL: https://github.com/apache/solr/pull/2828#discussion_r1825909291
##########
solr/solrj/src/test/org/apache/solr/client/solrj/impl/MockTrustManager.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.impl;
+
+import java.net.Socket;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLEngine;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509ExtendedTrustManager;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.util.SSLTestConfig;
+
+/**
+ * Taken from: https://www.baeldung.com/java-httpclient-ssl sec 4.1, 2024/02/12. This is an
+ * all-trusting Trust Manager. Works with self-signed certificates.
+ */
+public class MockTrustManager extends X509ExtendedTrustManager {
Review Comment:
Thank you for taking the time to review. I'll plan to commit this (main only?; 9.8?) early next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17516: LBHttpSolrClient: support HttpJdkSolrClient (Generic Version) [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #2828:
URL: https://github.com/apache/solr/pull/2828#discussion_r1825922973
##########
solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpJdkSolrClient.java:
##########
@@ -173,8 +173,8 @@ public NamedList<Object> request(SolrRequest<?> solrRequest, String collection)
"Timeout occurred while waiting response from server at: " + pReq.url, e);
} catch (SolrException se) {
throw se;
- } catch (RuntimeException re) {
- throw new SolrServerException(re);
+ } catch (IOException | RuntimeException e) {
Review Comment:
Okay; just needs a comment because it looks wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org