You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "TessaIO (via GitHub)" <gi...@apache.org> on 2025/12/12 17:29:47 UTC
[PR] feat: log total bytes gathered when max scatter-gather bytes limit is reached (druid)
TessaIO opened a new pull request, #18841:
URL: https://github.com/apache/druid/pull/18841
Fixes #18840.
<!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
<!-- If you are a committer, follow the PR action item checklist for committers:
https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
### Description
log total bytes gathered when max scatter-gather bytes limit is reached
#### Release note
<hr> log total bytes gathered when max scatter-gather bytes limit is reached
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [x] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [x] added documentation for new or modified features or behaviors.
- [x] a release note entry in the PR description.
- [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [x] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [x] added integration tests.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: log total bytes gathered when max scatter-gather bytes limit is reached (druid)
Posted by "TessaIO (via GitHub)" <gi...@apache.org>.
TessaIO commented on code in PR #18841:
URL: https://github.com/apache/druid/pull/18841#discussion_r2616425543
##########
server/src/main/java/org/apache/druid/client/DirectDruidClient.java:
##########
@@ -439,9 +439,10 @@ private void checkTotalBytesLimit(long bytes)
{
if (maxScatterGatherBytes < Long.MAX_VALUE && totalBytesGathered.addAndGet(bytes) > maxScatterGatherBytes) {
String msg = StringUtils.format(
- "Query[%s] url[%s] max scatter-gather bytes limit reached.",
+ "Query[%s] url[%s] max scatter-gather bytes limit reached. Total bytes gathered are equal to: %d",
query.getId(),
- url
+ url,
+ totalBytesGathered.get()
);
Review Comment:
The condition is updated by also removing `maxScatterGatherBytes < Long.MAX_VALUE` as the condition `currentTotalBytesGathered > maxScatterGatherBytes` already covers it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: log total bytes gathered when max scatter-gather bytes limit is reached (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #18841:
URL: https://github.com/apache/druid/pull/18841#discussion_r2625119291
##########
server/src/main/java/org/apache/druid/client/DirectDruidClient.java:
##########
@@ -437,11 +437,14 @@ private long checkQueryTimeout()
private void checkTotalBytesLimit(long bytes)
{
- if (maxScatterGatherBytes < Long.MAX_VALUE && totalBytesGathered.addAndGet(bytes) > maxScatterGatherBytes) {
+ final long currentTotalBytesGathered = totalBytesGathered.addAndGet(bytes);
+ if (currentTotalBytesGathered > maxScatterGatherBytes) {
String msg = StringUtils.format(
- "Query[%s] url[%s] max scatter-gather bytes limit reached.",
+ "Query[%s] url[%s] total bytes gathered[%,d] exceeds maxScatterGatherBytes[%,d]",
query.getId(),
- url
+ url,
+ maxScatterGatherBytes,
+ currentTotalBytesGathered
Review Comment:
@TessaIO the arguments need to be swapped to match the error message order - total bytes first, then max total bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: log total bytes gathered when max scatter-gather bytes limit is reached (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #18841:
URL: https://github.com/apache/druid/pull/18841#discussion_r2615911648
##########
server/src/main/java/org/apache/druid/client/DirectDruidClient.java:
##########
@@ -439,9 +439,10 @@ private void checkTotalBytesLimit(long bytes)
{
if (maxScatterGatherBytes < Long.MAX_VALUE && totalBytesGathered.addAndGet(bytes) > maxScatterGatherBytes) {
String msg = StringUtils.format(
- "Query[%s] url[%s] max scatter-gather bytes limit reached.",
+ "Query[%s] url[%s] max scatter-gather bytes limit reached. Total bytes gathered are equal to: %d",
query.getId(),
- url
+ url,
+ totalBytesGathered.get()
);
Review Comment:
We can assign the result of total bytes gathered by this query and also log `maxScatterGatherBytes` while at it. Something like:
```suggestion
final long currentTotalBytesGathered = totalBytesGathered.addAndGet(bytes);
if (maxScatterGatherBytes < Long.MAX_VALUE && currentTotalBytesGathered > maxScatterGatherBytes) {
String msg = StringUtils.format(
"Query[%s] url[%s] maxScatterGatherBytes[%,d] limit reached. Total bytes gathered: [%,d]",
query.getId(),
url,
maxScatterGatherBytes,
currentTotalBytesGathered
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: log total bytes gathered when max scatter-gather bytes limit is reached (druid)
Posted by "TessaIO (via GitHub)" <gi...@apache.org>.
TessaIO commented on PR #18841:
URL: https://github.com/apache/druid/pull/18841#issuecomment-3649638652
@abhishekrb19 thanks for the suggestion, I think it would be cleaner if the test coverage is addressed in a separate PR as this one is solely focused on improving the logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: log total bytes gathered when max scatter-gather bytes limit is reached (druid)
Posted by "TessaIO (via GitHub)" <gi...@apache.org>.
TessaIO commented on PR #18841:
URL: https://github.com/apache/druid/pull/18841#issuecomment-3662925095
@abhishekrb19 thanks for the review, updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: log total bytes gathered when max scatter-gather bytes limit is reached (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #18841:
URL: https://github.com/apache/druid/pull/18841#discussion_r2625125085
##########
server/src/test/java/org/apache/druid/query/QueryRunnerBasedOnClusteredClientTestBase.java:
##########
@@ -228,9 +228,7 @@ protected static List<Result<TimeseriesResultValue>> expectedTimeseriesResult(in
protected static ResponseContext responseContext()
{
- final ResponseContext responseContext = ConcurrentResponseContext.createEmpty();
- responseContext.initializeRemainingResponses();
- return responseContext;
+ return DirectDruidClient.makeResponseContextForQuery();
Review Comment:
could you address the checkstyle unused import errors?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: log total bytes gathered when max scatter-gather bytes limit is reached (druid)
Posted by "TessaIO (via GitHub)" <gi...@apache.org>.
TessaIO commented on code in PR #18841:
URL: https://github.com/apache/druid/pull/18841#discussion_r2625201799
##########
server/src/main/java/org/apache/druid/client/DirectDruidClient.java:
##########
@@ -437,11 +437,14 @@ private long checkQueryTimeout()
private void checkTotalBytesLimit(long bytes)
{
- if (maxScatterGatherBytes < Long.MAX_VALUE && totalBytesGathered.addAndGet(bytes) > maxScatterGatherBytes) {
+ final long currentTotalBytesGathered = totalBytesGathered.addAndGet(bytes);
+ if (currentTotalBytesGathered > maxScatterGatherBytes) {
String msg = StringUtils.format(
- "Query[%s] url[%s] max scatter-gather bytes limit reached.",
+ "Query[%s] url[%s] total bytes gathered[%,d] exceeds maxScatterGatherBytes[%,d]",
query.getId(),
- url
+ url,
+ maxScatterGatherBytes,
+ currentTotalBytesGathered
Review Comment:
thanks for spotting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: log total bytes gathered when max scatter-gather bytes limit is reached (druid)
Posted by "TessaIO (via GitHub)" <gi...@apache.org>.
TessaIO commented on PR #18841:
URL: https://github.com/apache/druid/pull/18841#issuecomment-3663059890
@abhishekrb19 done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: log total bytes gathered when max scatter-gather bytes limit is reached (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #18841:
URL: https://github.com/apache/druid/pull/18841#discussion_r2620759570
##########
server/src/main/java/org/apache/druid/client/DirectDruidClient.java:
##########
@@ -437,11 +437,13 @@ private long checkQueryTimeout()
private void checkTotalBytesLimit(long bytes)
{
- if (maxScatterGatherBytes < Long.MAX_VALUE && totalBytesGathered.addAndGet(bytes) > maxScatterGatherBytes) {
+ final long currentTotalBytesGathered = totalBytesGathered.addAndGet(bytes);
+ if (currentTotalBytesGathered > maxScatterGatherBytes) {
String msg = StringUtils.format(
- "Query[%s] url[%s] max scatter-gather bytes limit reached.",
+ "Query[%s] url[%s] max scatter-gather bytes limit reached. Total bytes gathered: %d",
query.getId(),
- url
+ url,
+ currentTotalBytesGathered
);
Review Comment:
While at it, let's also include the `maxScatterGatherBytes` value in the exception message.
nit: use `[]` to interpolate per Druid [style-guide](https://github.com/apache/druid/blob/master/dev/style-conventions.md#message-formatting-for-logs-and-exceptions).
```
Query[%s] url[%s] total bytes gathered[%,d] exceeds maxScatterGatherBytes[%,d]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] feat: log total bytes gathered when max scatter-gather bytes limit is reached (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 merged PR #18841:
URL: https://github.com/apache/druid/pull/18841
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org