You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "waterWang (via GitHub)" <gi...@apache.org> on 2026/08/24 21:09:26 UTC
[PR] fix: Make HttpLoadQueuePeon degrade gracefully instead of throwing on fetch failure (druid)
waterWang opened a new pull request, #20128:
URL: https://github.com/apache/druid/pull/20128
### Description
Fixes #19950
When a historical server returns a non-200/404 response or is unreachable
during `fetchSegmentLoadingCapabilities()`, the `HttpLoadQueuePeon` constructor
throws an `RE`. This exception propagates through
`LoadQueueTaskMaster.resetPeonsForNewServers()` into
`PrepareBalancerAndLoadQueues`, which is the first duty in the
`HistoricalManagementDuties` group. The top-level
`DruidCoordinator.DutiesRunnable.run()` catches and logs the exception, but
segment management (loading, balancing, handoffs) stops entirely for **all**
servers — not just the unhealthy one. Ingestion tasks back up waiting for
handoff.
### Changes
- `fetchSegmentLoadingCapabilities()`: Instead of throwing `RE` on non-200/404
responses or any `Throwable`, return default `SegmentLoadingCapabilities`
derived from the configured batch size, with a warning log. The peon is still
created, the server is still managed with conservative defaults, and the rest
of the duty group proceeds normally.
- Mirror the existing 404 branch's fallback behavior (already returned default
capabilities).
### Key design decisions
- The 404 branch already existed and returned default capabilities. The fix
extends this same graceful degradation to non-200 responses and exceptions.
- `log.makeAlert()` is preserved for non-200 responses so operators still get
alerted about the unhealthy server.
- The catch block now logs a warning instead of throwing, keeping the
Coordinator ticking.
### Release notes
Fixed a bug where a single unhealthy or unreachable historical server could
stall all segment loading and balancing across the cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: Make HttpLoadQueuePeon degrade gracefully instead of throwing on fetch failure (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on PR #20128:
URL: https://github.com/apache/druid/pull/20128#issuecomment-5413580997
> @waterWang , does this mean we don't need #19550 ? cc: @abhishekrb19
@waterWang since https://github.com/apache/druid/pull/19950 already addresses the bug and has now been merged, we can close this one out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: Make HttpLoadQueuePeon degrade gracefully instead of throwing on fetch failure (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 closed pull request #20128: fix: Make HttpLoadQueuePeon degrade gracefully instead of throwing on fetch failure
URL: https://github.com/apache/druid/pull/20128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: Make HttpLoadQueuePeon degrade gracefully instead of throwing on fetch failure (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #20128:
URL: https://github.com/apache/druid/pull/20128#issuecomment-5411206745
@waterWang , does this mean we don't need #19550 ?
cc: @abhishekrb19
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: Make HttpLoadQueuePeon degrade gracefully instead of throwing on fetch failure (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #20128:
URL: https://github.com/apache/druid/pull/20128#discussion_r3853025159
##########
server/src/main/java/org/apache/druid/server/coordinator/loading/HttpLoadQueuePeon.java:
##########
@@ -187,7 +193,9 @@ private SegmentLoadingCapabilities fetchSegmentLoadingCapabilities()
);
}
catch (Throwable th) {
- throw new RE(th, "Received error while fetching historical capabilities from Server[%s].", serverId);
+ log.warn(th, "Failed to fetch loading capabilities from server[%s]. Using default capabilities.", serverId);
Review Comment:
[P2] Fallback swallows fatal and interruption signals
The Throwable catch now converts fatal Errors and InterruptedException into a normal peon. This can mask JVM-level failures and clear interruption without restoring the thread flag. Catch only expected request/parsing exceptions, while rethrowing fatal errors and preserving interruption.
##########
server/src/main/java/org/apache/druid/server/coordinator/loading/HttpLoadQueuePeon.java:
##########
@@ -187,7 +193,9 @@ private SegmentLoadingCapabilities fetchSegmentLoadingCapabilities()
);
}
catch (Throwable th) {
- throw new RE(th, "Received error while fetching historical capabilities from Server[%s].", serverId);
+ log.warn(th, "Failed to fetch loading capabilities from server[%s]. Using default capabilities.", serverId);
+ int batchSize = config.getBatchSize() == null ? 1 : config.getBatchSize();
+ return new SegmentLoadingCapabilities(batchSize, batchSize);
Review Comment:
[P2] Transient failures permanently throttle the peon
serverCapabilities is initialized once and cached for the peon's lifetime. After a temporary timeout, 5xx, or malformed response, the stored fallback remains active indefinitely; with no configured batchSize this reduces normal and turbo loading to one segment per batch. Retry capability discovery or refresh the fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org