You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "gerlowskija (via GitHub)" <gi...@apache.org> on 2024/08/06 19:52:17 UTC
[PR] SOLR-17394: Check response status in IndexFetcher [solr]
gerlowskija opened a new pull request, #2621:
URL: https://github.com/apache/solr/pull/2621
https://issues.apache.org/jira/browse/SOLR-17394
# Description
Prior to this commit, IndexFetcher would mis-read 404 responses from other nodes. It eventually catches this mistake and retries, but not before it wastefully allocates a massive (> 1GB) byte array.
These allocations were relatively infrequent, as they require out of date leader information to occur. But they are big enough to impact performance in production, and were observed to cause OOM's when running tests.
# Solution
This commit avoids this by checking the HTTP status code of all IndexFetcher requests/responses, even those that utilize InputStreamResponseParser.
# Tests
I don't have unit tests for this as it's a bit hard to exercise in isolation. But I am able to see the newly-introduced IOE message in the logs, and the massive `byte[]` is no longer allocated.
The other way I've tested this, such as it is, is through 'beasting' a particular test which trigger a OOM and heap dump on about every 1/10 iterations. Setting `tests.heapsize` to any moderate value (e.g. `1GB`) in `gradle.properties` and running the command below reproduces the OOM reliably for me on `main` without this PR. (And reproduces the OOM never with the PR in place.)
```
./gradlew -p solr/solrj beast -Ptests.seed=F024B5067D7FCE8F -Ptests.neverUpToDate=true -Ptests.dups=50 --tests CloudHttp2SolrClientTest
```
# Checklist
Please review the following and check all that apply:
- [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability.
- [x] I have created a Jira issue and added the issue ID to my pull request title.
- [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [x] I have developed this patch against the `main` branch.
- [x] I have run `./gradlew check`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17394: Check response status in IndexFetcher [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #2621:
URL: https://github.com/apache/solr/pull/2621#discussion_r1709937216
##########
solr/core/src/java/org/apache/solr/handler/IndexFetcher.java:
##########
@@ -1988,16 +1989,38 @@ private FastInputStream getStream() throws IOException {
req.setBasePath(leaderBaseUrl);
if (useExternalCompression) req.addHeader("Accept-Encoding", "gzip");
response = solrClient.request(req, leaderCoreName);
+ final var responseStatus = (Integer) response.get("responseStatus");
is = (InputStream) response.get("stream");
+
+ if (responseStatus != 200) {
+ final var errorMsg =
+ String.format(
+ Locale.ROOT,
+ "Unexpected status code [%d] when downloading file [%s].",
+ responseStatus,
+ fileName);
+ closeStreamAndThrowIOE(is, errorMsg, Optional.empty());
+ }
+
if (useInternalCompression) {
is = new InflaterInputStream(is);
}
return new FastInputStream(is);
} catch (Exception e) {
- // close stream on error
- IOUtils.closeQuietly(is);
- throw new IOException("Could not download file '" + fileName + "'", e);
+ closeStreamAndThrowIOE(is, "Could not download file '" + fileName + "'", Optional.of(e));
+ }
+
+ // Unreachable b/c closeStreamAndThrowIOE will always throw
Review Comment:
You could have returned the IOException and let the caller throw it.
##########
solr/core/src/java/org/apache/solr/handler/IndexFetcher.java:
##########
@@ -1988,16 +1989,38 @@ private FastInputStream getStream() throws IOException {
req.setBasePath(leaderBaseUrl);
if (useExternalCompression) req.addHeader("Accept-Encoding", "gzip");
response = solrClient.request(req, leaderCoreName);
+ final var responseStatus = (Integer) response.get("responseStatus");
is = (InputStream) response.get("stream");
+
+ if (responseStatus != 200) {
+ final var errorMsg =
+ String.format(
+ Locale.ROOT,
+ "Unexpected status code [%d] when downloading file [%s].",
+ responseStatus,
+ fileName);
+ closeStreamAndThrowIOE(is, errorMsg, Optional.empty());
+ }
+
if (useInternalCompression) {
is = new InflaterInputStream(is);
}
return new FastInputStream(is);
} catch (Exception e) {
- // close stream on error
- IOUtils.closeQuietly(is);
- throw new IOException("Could not download file '" + fileName + "'", e);
+ closeStreamAndThrowIOE(is, "Could not download file '" + fileName + "'", Optional.of(e));
+ }
+
+ // Unreachable b/c closeStreamAndThrowIOE will always throw
+ return null;
+ }
+
+ private void closeStreamAndThrowIOE(
+ InputStream is, String exceptionMessage, Optional<Exception> e) throws IOException {
Review Comment:
A matter of taste but I would prefer we/programmers use Optionals conservatively -- when it's clearly a help. They are verbose; should have been built into the language like Kotlin. In particular, I suggest we not use Optional as a parameter; only return them if we use them.
See https://www.baeldung.com/java-optional #15 (and I've seen this advice elsewhere too)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17394: Check response status in IndexFetcher [solr]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on PR #2621:
URL: https://github.com/apache/solr/pull/2621#issuecomment-2292530594
> This was merged so fast that it got not review; I wanted to review it
I thought 3 days was a reasonable length of time to wait on a pretty clear and minor bugfix; I hope i'm not ruffling any feathers by merging on that timeframe.
But, that said, I'm more than happy to take the review comments now, since you didn't get a chance to review beforehand. I've replied to several comments "in-line", and will push fixes to address.
(FYI @dsmiley )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17394: Check response status in IndexFetcher [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #2621:
URL: https://github.com/apache/solr/pull/2621#discussion_r1719667781
##########
solr/core/src/java/org/apache/solr/handler/IndexFetcher.java:
##########
@@ -1988,16 +1989,38 @@ private FastInputStream getStream() throws IOException {
req.setBasePath(leaderBaseUrl);
if (useExternalCompression) req.addHeader("Accept-Encoding", "gzip");
response = solrClient.request(req, leaderCoreName);
+ final var responseStatus = (Integer) response.get("responseStatus");
is = (InputStream) response.get("stream");
+
+ if (responseStatus != 200) {
+ final var errorMsg =
+ String.format(
+ Locale.ROOT,
+ "Unexpected status code [%d] when downloading file [%s].",
+ responseStatus,
+ fileName);
+ closeStreamAndThrowIOE(is, errorMsg, Optional.empty());
+ }
+
if (useInternalCompression) {
is = new InflaterInputStream(is);
}
return new FastInputStream(is);
} catch (Exception e) {
- // close stream on error
- IOUtils.closeQuietly(is);
- throw new IOException("Could not download file '" + fileName + "'", e);
+ closeStreamAndThrowIOE(is, "Could not download file '" + fileName + "'", Optional.of(e));
+ }
+
+ // Unreachable b/c closeStreamAndThrowIOE will always throw
+ return null;
+ }
+
+ private void closeStreamAndThrowIOE(
+ InputStream is, String exceptionMessage, Optional<Exception> e) throws IOException {
Review Comment:
We really need more coding style guides and ideally lint checking tools for things like Optional. Or in 12 months, we'll have this same conversation, but with someone new ;-(.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17394: Check response status in IndexFetcher [solr]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #2621:
URL: https://github.com/apache/solr/pull/2621#discussion_r1719161093
##########
solr/core/src/java/org/apache/solr/handler/IndexFetcher.java:
##########
@@ -1988,16 +1989,38 @@ private FastInputStream getStream() throws IOException {
req.setBasePath(leaderBaseUrl);
if (useExternalCompression) req.addHeader("Accept-Encoding", "gzip");
response = solrClient.request(req, leaderCoreName);
+ final var responseStatus = (Integer) response.get("responseStatus");
is = (InputStream) response.get("stream");
+
+ if (responseStatus != 200) {
+ final var errorMsg =
+ String.format(
+ Locale.ROOT,
+ "Unexpected status code [%d] when downloading file [%s].",
+ responseStatus,
+ fileName);
+ closeStreamAndThrowIOE(is, errorMsg, Optional.empty());
+ }
+
if (useInternalCompression) {
is = new InflaterInputStream(is);
}
return new FastInputStream(is);
} catch (Exception e) {
- // close stream on error
- IOUtils.closeQuietly(is);
- throw new IOException("Could not download file '" + fileName + "'", e);
+ closeStreamAndThrowIOE(is, "Could not download file '" + fileName + "'", Optional.of(e));
+ }
+
+ // Unreachable b/c closeStreamAndThrowIOE will always throw
+ return null;
+ }
+
+ private void closeStreamAndThrowIOE(
+ InputStream is, String exceptionMessage, Optional<Exception> e) throws IOException {
Review Comment:
I've seen the critique of `Optionals` before, but I'll admit it doesn't makes a lot of sense to me.
**If** your only goal in using an `Optional` is to avoid a null-check, **and** you want to be absolutely rigid on sanity-checking your inputs...then sure, I guess taking an `Optional` parameter doesn't help you.
But it has other benefits - among them its self-documenting nature. It signals to readers unequivocally that this parameter doesn't need a value, strictly speaking. IMO `Optional` used in this way is like a better version of the `@Nullable` annotation - "better" because `@Nullable` only serves as a reminder for those reading the method signature itself, whereas `Optional` in its role as conduit to the underlying value manages to reinforce this reminder throughout the method implementation.
That said - I understand I'm a minority opinion here, so I'm happy to defer to the Java community's consensus on this. I'll try to remember this and not introduce further usage in this vein.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17394: Check response status in IndexFetcher [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #2621:
URL: https://github.com/apache/solr/pull/2621#issuecomment-2292801661
I see less than 2 days: August 6th 3:52pm -> 8th 8:32am. That's too short. Also we generally give advanced merge notice if there was no feedback whatsoever.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17394: Check response status in IndexFetcher [solr]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija merged PR #2621:
URL: https://github.com/apache/solr/pull/2621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17394: Check response status in IndexFetcher [solr]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #2621:
URL: https://github.com/apache/solr/pull/2621#discussion_r1719161279
##########
solr/core/src/java/org/apache/solr/handler/IndexFetcher.java:
##########
@@ -1988,16 +1989,38 @@ private FastInputStream getStream() throws IOException {
req.setBasePath(leaderBaseUrl);
if (useExternalCompression) req.addHeader("Accept-Encoding", "gzip");
response = solrClient.request(req, leaderCoreName);
+ final var responseStatus = (Integer) response.get("responseStatus");
is = (InputStream) response.get("stream");
+
+ if (responseStatus != 200) {
+ final var errorMsg =
+ String.format(
+ Locale.ROOT,
+ "Unexpected status code [%d] when downloading file [%s].",
+ responseStatus,
+ fileName);
+ closeStreamAndThrowIOE(is, errorMsg, Optional.empty());
+ }
+
if (useInternalCompression) {
is = new InflaterInputStream(is);
}
return new FastInputStream(is);
} catch (Exception e) {
- // close stream on error
- IOUtils.closeQuietly(is);
- throw new IOException("Could not download file '" + fileName + "'", e);
+ closeStreamAndThrowIOE(is, "Could not download file '" + fileName + "'", Optional.of(e));
+ }
+
+ // Unreachable b/c closeStreamAndThrowIOE will always throw
Review Comment:
Yeah, that would've been better in hindsight, good point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org