You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "jtuglu1 (via GitHub)" <gi...@apache.org> on 2025/09/21 21:19:55 UTC
[PR] Detect segment unavailability for queries in broker (druid)
jtuglu1 opened a new pull request, #18558:
URL: https://github.com/apache/druid/pull/18558
Refactor of https://github.com/apache/druid/pull/13967/
<!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
<!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
Fixes #XXXX.
<!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
<!-- If you are a committer, follow the PR action item checklist for committers:
https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
### Description
<!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
<!-- Describe your patch: what did you change in code? How did you fix the problem? -->
<!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
#### Fixed the bug ...
#### Renamed the class ...
#### Added a forbidden-apis entry ...
<!--
In each section, please describe design decisions made, including:
- Choice of algorithms
- Behavioral aspects. What configuration values are acceptable? How are corner cases and error conditions handled, such as when there are insufficient resources?
- Class organization and design (how the logic is split between classes, inheritance, composition, design patterns)
- Method organization and design (how the logic is split between methods, parameters and return types)
- Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
-->
<!-- It's good to describe an alternative design (or mention an alternative name) for every design (or naming) decision point and compare the alternatives with the designs that you've implemented (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
<!-- If there was a discussion of the design of the feature implemented in this PR elsewhere (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), link to that discussion from this PR description and explain what have changed in your final design compared to your original proposal or the consensus version in the end of the discussion. If something hasn't changed since the original discussion, you can omit a detailed discussion of those aspects of the design here, perhaps apart from brief mentioning for the sake of readability of this PR description. -->
<!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
#### Release note
<!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users.
If your change doesn't have end user impact, you can skip this section.
For tips about how to write a good release note, see [Release notes](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#release-notes).
-->
<hr>
##### Key changed/added classes in this PR
* `MyFoo`
* `OurBar`
* `TheirBaz`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [WIP]: Detect segment unavailability for queries in broker (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18558:
URL: https://github.com/apache/druid/pull/18558#issuecomment-3829868021
This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [WIP]: Detect segment unavailability for queries in broker (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18558:
URL: https://github.com/apache/druid/pull/18558#issuecomment-3707479856
This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [WIP]: Detect segment unavailability for queries in broker (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on PR #18558:
URL: https://github.com/apache/druid/pull/18558#issuecomment-3482490205
> @jtuglu1 , I haven't gone through the full approach in the PR yet. But had a small comment on the flowchart - a Broker should start serving a segment only after it has received information from the Coordinator that the segment is used and has been loaded on any data node atleast once. I think you might have meant to capture this but the flowchart only mentions a "Coordinator sync USED callback" instead of "Coordinator sync USED=true + HAS_LOADED=true callback".
>
> For reference, see [#13967 (comment)](https://github.com/apache/druid/pull/13967#sync-issues), item 2b in the original PR.
Yes – I think the initial goal of the diagram was to hide the complexity of how the queryable property was actually set, and focus on the decision flow w.r.t the broker timeline. I'll extend the diagram to include the full state transfer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [WIP]: Detect segment unavailability for queries in broker (druid)
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18558:
URL: https://github.com/apache/druid/pull/18558#discussion_r2480180080
##########
server/src/main/java/org/apache/druid/client/CachingClusteredClient.java:
##########
@@ -476,15 +486,44 @@ private Set<SegmentServerSelector> computeSegmentsToQuery(
ServerSelector server = chunk.getObject();
if (isRealtimeSegmentOnly && !server.isRealtimeSegment()) {
continue; // Skip historical segments when only realtime segments are requested
+ } else if (brokerSegmentWatcherConfig.detectUnavailableSegments() && !server.isQueryable()) {
+ log.debug("ServerSelector for segment id [%s] is not queryable", server.getSegment().getId());
+ continue;
}
+
final SegmentDescriptor segment = new SegmentDescriptor(
holder.getInterval(),
holder.getVersion(),
chunk.getChunkNumber()
);
segments.add(new SegmentServerSelector(server, segment));
+ if (server.isEmpty()) {
+ unavailableSegmentsIds.add(server.getSegment().getId());
+ }
+ }
+ }
+
+ if (brokerSegmentWatcherConfig.detectUnavailableSegments() && !unavailableSegmentsIds.isEmpty()) {
+ queryPlus.getQueryMetrics().reportUnavailableSegmentCount(unavailableSegmentsIds.size()).emit(emitter);
+ log.warn(
+ "Detected [%d] unavailable segments, trimmed segment ids: [%s]",
+ unavailableSegmentsIds.size(),
+ unavailableSegmentsIds.subList(0, 10)
+ );
+ if (unavailableSegmentsAction == UnavailableSegmentsAction.FAIL) {
+ throw new QueryException(
+ QueryException.UNAVAILABLE_SEGMENTS_ERROR_CODE,
+ StringUtils.format(
+ "Detected [%d] unavailable segments, trimmed segment ids: [%s]",
+ unavailableSegmentsIds.size(),
+ unavailableSegmentsIds.subList(0, 100)
Review Comment:
Same issue as the previous comment - the code doesn't validate that the list size is at least 100 before calling `subList(0, 100)`. Consider using `Math.min(100, unavailableSegmentsIds.size())` to avoid potential IndexOutOfBoundsException.
```suggestion
unavailableSegmentsIds.subList(0, Math.min(100, unavailableSegmentsIds.size()))
```
##########
server/src/main/java/org/apache/druid/client/BrokerServerView.java:
##########
@@ -56,7 +62,61 @@
import java.util.stream.Collectors;
/**
- *
+ * <p>
+ * Maintains a timeline of segments per datasource.
+ * This timeline is populated by callback received from datanodes when they load a segment
+ * or when realtime segments are created in a task.
+ * Downstream classes can also register timeline callback on this class, for example BrokerSegmentMetadataCache.
+ * </p>
+ * <p>
+ * There is a second flow which is enabled only when unavailabe segment detection is turned on.
Review Comment:
Corrected spelling of 'unavailabe' to 'unavailable'.
```suggestion
* There is a second flow which is enabled only when unavailable segment detection is turned on.
```
##########
server/src/main/java/org/apache/druid/server/http/MetadataResource.java:
##########
@@ -201,6 +206,138 @@ public Response getAllUsedSegments(
}
}
+ /**
+ * <p>This endpoint is used by MetadataSegmentView in broker to keep an up-to-date list of segments present in the system.
+ * This endpoint lists segments present in the system and can also incrementally provide the segments added/dropped
+ * since last response.</p>
+ * <br>Flow
+ * <ol>
+ * <li>Client sends first request /druid/coordinator/v1/metadata/changedSegments?counter=-1
+ * Server responds with list of segments currently present and a <counter,hash> pair. </li>
+ * <li>Client sends subsequent requests /druid/coordinator/v1/metadata/changedSegments?counter=<counter>&hash=<hash>
+ * Where <counter,hash> values are used from the last response. Server responds with list of segment updates
+ * since given counter.</li>
+ * </ol>
+ *
+ * @param req request
+ * @param dataSources requested datasources
+ * @param counter counter received in last response.
+ * @param hash hash received in last response.
+ */
+ @GET
+ @Path("/changedSegments")
+ @Produces(MediaType.APPLICATION_JSON)
+ public Response getChangedSegments(
+ @Context final HttpServletRequest req,
+ @QueryParam("datasources") final @Nullable Set<String> dataSources,
+ @QueryParam("counter") long counter,
+ @QueryParam("hash") long hash
+ )
+ {
+ Set<String> requiredDataSources = (null == dataSources) ? new HashSet<>() : dataSources;
+
+ log.debug(
+ "Changed segments requested. counter [%d], hash [%d], dataSources [%s]",
+ counter,
+ hash,
+ requiredDataSources
+ );
+
+ DataSourcesSnapshot dataSourcesSnapshot = segmentsMetadataManager.getSnapshotOfDataSourcesWithAllUsedSegments();
+ ChangeRequestHistory<List<DataSegmentChange>> changeRequestHistory = segmentsMetadataManager.getChangeRequestHistory();
+
+ ChangeRequestsSnapshot<List<DataSegmentChange>> changeRequestsSnapshot = changeRequestHistory.getRequestsSinceSync(
+ new ChangeRequestHistory.Counter(counter, hash));
+ List<List<DataSegmentChange>> requests = changeRequestsSnapshot.getRequests();
+ List<DataSegmentChange> flatRequests = new ArrayList<>();
+ if (null != requests) {
+ requests.forEach(flatRequests::addAll);
+ }
+
+ List<DataSegmentChange> dataSegmentChanges;
+ ChangeRequestHistory.Counter lastCounter = changeRequestsSnapshot.getCounter();
+ boolean reset = false;
+ String resetCause = "";
+ if (changeRequestsSnapshot.isResetCounter()) {
+ reset = true;
+ dataSegmentChanges =
+ dataSourcesSnapshot
+ .getDataSourcesWithAllUsedSegments()
+ .stream()
+ .flatMap(druidDataSource -> druidDataSource.getSegments().stream())
+ .filter(segment -> requiredDataSources.isEmpty()
+ || requiredDataSources.contains(segment.getDataSource()))
+ .map(segment -> {
+ Long numRows = null;
+ if (coordinatorSegmentMetadataCache != null) {
+ AvailableSegmentMetadata availableSegmentMetadata = coordinatorSegmentMetadataCache.getAvailableSegmentMetadata(
+ segment.getDataSource(),
+ segment.getId()
+ );
+ if (null != availableSegmentMetadata) {
+ numRows = availableSegmentMetadata.getNumRows();
+ }
+ }
+ return new DataSegmentChange(
+ new SegmentStatusInCluster(
+ segment,
+ dataSourcesSnapshot.getOvershadowedSegments().contains(segment),
+ null,
+ numRows,
+ false, //
Review Comment:
Incomplete inline comment. The comment appears to be cut off or empty. Either complete the comment to explain why this parameter is false, or remove the comment marker.
```suggestion
false,
```
##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataManager.java:
##########
@@ -804,13 +856,57 @@ private void createDatasourcesSnapshot(DateTime snapshotTime, List<DataSegment>
// Updates outside of database polls were primarily for the user experience, so users would immediately see the
// effect of a segment mark call reflected in MetadataResource API calls.
- dataSourcesSnapshot = DataSourcesSnapshot.fromUsedSegments(
- Iterables.filter(segments, Objects::nonNull), // Filter corrupted entries (see above in this method).
- snapshotTime
- );
- emitMetric("segment/buildSnapshot/time", stopwatch.millisElapsed());
- log.debug(
- "Created snapshot from polled segments in [%d]ms. Found [%d] overshadowed segments.",
+ if (segments.isEmpty()) {
+ log.debug("No segments found in the database!");
+ } else {
+ log.info("Polled and found %,d segments in the database", segments.size());
+ }
Review Comment:
[nitpick] The empty segment check and logging were added here, but this duplicates the logging that occurs after the snapshot is created at line 908. Consider consolidating the logging to avoid redundancy.
```suggestion
// Removed redundant logging of segment count; snapshot creation log below suffices.
```
##########
server/src/main/java/org/apache/druid/client/CachingClusteredClient.java:
##########
@@ -476,15 +486,44 @@ private Set<SegmentServerSelector> computeSegmentsToQuery(
ServerSelector server = chunk.getObject();
if (isRealtimeSegmentOnly && !server.isRealtimeSegment()) {
continue; // Skip historical segments when only realtime segments are requested
+ } else if (brokerSegmentWatcherConfig.detectUnavailableSegments() && !server.isQueryable()) {
+ log.debug("ServerSelector for segment id [%s] is not queryable", server.getSegment().getId());
+ continue;
}
+
final SegmentDescriptor segment = new SegmentDescriptor(
holder.getInterval(),
holder.getVersion(),
chunk.getChunkNumber()
);
segments.add(new SegmentServerSelector(server, segment));
+ if (server.isEmpty()) {
+ unavailableSegmentsIds.add(server.getSegment().getId());
+ }
+ }
+ }
+
+ if (brokerSegmentWatcherConfig.detectUnavailableSegments() && !unavailableSegmentsIds.isEmpty()) {
+ queryPlus.getQueryMetrics().reportUnavailableSegmentCount(unavailableSegmentsIds.size()).emit(emitter);
+ log.warn(
+ "Detected [%d] unavailable segments, trimmed segment ids: [%s]",
+ unavailableSegmentsIds.size(),
+ unavailableSegmentsIds.subList(0, 10)
Review Comment:
The code doesn't validate that the list size is at least 10 before calling `subList(0, 10)`, which could throw an IndexOutOfBoundsException if there are fewer than 10 unavailable segments. Consider using `Math.min(10, unavailableSegmentsIds.size())` instead.
##########
sql/src/test/java/org/apache/druid/sql/calcite/schema/SystemSchemaTest.java:
##########
@@ -682,11 +684,11 @@ public void testSegmentsTableWithProjection() throws JsonProcessingException
{
final SegmentsTable segmentsTable = new SegmentsTable(druidSchema, metadataView, MAPPER, authMapper);
final Set<SegmentStatusInCluster> publishedSegments = new HashSet<>(Arrays.asList(
- new SegmentStatusInCluster(publishedCompactedSegment1, true, 2, null, false),
- new SegmentStatusInCluster(publishedCompactedSegment2, false, 0, null, false),
- new SegmentStatusInCluster(publishedUncompactedSegment3, false, 2, null, false),
- new SegmentStatusInCluster(segment1, true, 2, null, false),
- new SegmentStatusInCluster(segment2, false, 0, null, false)
+ new SegmentStatusInCluster(publishedCompactedSegment1, true, 2, null, false,false),
Review Comment:
Missing space after comma between the two boolean parameters. Should be `false, false` for consistency with the other lines in this test file.
```suggestion
new SegmentStatusInCluster(publishedCompactedSegment1, true, 2, null, false, false),
```
##########
server/src/main/java/org/apache/druid/client/BrokerServerView.java:
##########
@@ -56,7 +62,61 @@
import java.util.stream.Collectors;
/**
- *
+ * <p>
+ * Maintains a timeline of segments per datasource.
+ * This timeline is populated by callback received from datanodes when they load a segment
+ * or when realtime segments are created in a task.
+ * Downstream classes can also register timeline callback on this class, for example BrokerSegmentMetadataCache.
+ * </p>
+ * <p>
+ * There is a second flow which is enabled only when unavailabe segment detection is turned on.
+ * {@link MetadataSegmentView} polls published segment metadata from the Coordinator. This class listens for published
+ * segment updates.
+ * The segment which are marked as `loaded` (used segment with non-zero replication factor which
+ * has been once loaded onto some historical) is added to the timeline. This is basically the set of segments that
+ * should be available for querying.
+ * </p>
+ * <p>
+ * Lifecycle for a segment s,
+ * <br>- s is created at time t1, ie. metadata for it is published in the database at time t1
+ * <br>- coordinator polls it at time t2, at this point the segment s is not considered as loaded
+ * <br>- broker polls coordinator at time t3 and finds segment s, but it is not added to the timeline since it is not loaded
+ * <br>- historical loads s at time t4
+ * <br>- coordinator and broker both receives callback from the historical
+ * <br>- coordinator marks the segment as `loaded` at time t5
+ * </p>
+ * now, lets consider two possibilities,
+ * <ol>
+ * <li>Broker receives segment metadata for s from coordinator before the callback from historical
+ * <ul>
+ * <li>
+ * broker adds s to the timeline at t6 with `queryable` field set to true,
+ * at this point any query that uses s would find that s is unavailable
+ * </li>
+ * <li>
+ * broker recieves callback for s from some historical at t7,
Review Comment:
Corrected spelling of 'recieves' to 'receives'.
```suggestion
* broker receives callback for s from some historical at t7,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [WIP]: Detect segment unavailability for queries in broker (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18558:
URL: https://github.com/apache/druid/pull/18558#issuecomment-3478765218
@jtuglu1 , I haven't gone through the full approach in the PR yet.
But had a small comment on the flowchart - a Broker should start serving a segment only after it has received information from the Coordinator that the segment is used and has been loaded on any data node atleast once. I think you might have meant to capture this but the flowchart only mentions a "Coordinator sync USED callback" instead of "Coordinator sync USED=1 + HAS_LOADED=1 callback".
For reference, see https://github.com/apache/druid/pull/13967#sync-issues, item 2b.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [WIP]: Detect segment unavailability for queries in broker (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18558:
URL: https://github.com/apache/druid/pull/18558#issuecomment-3471384800
@jtuglu1 , please ignore the Copilot review. Just triggered it to get an overall idea of the changes going in the PR.
As I understand from our offline discussion, the approach itself is still being finalized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] [WIP]: Detect segment unavailability for queries in broker (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #18558: [WIP]: Detect segment unavailability for queries in broker
URL: https://github.com/apache/druid/pull/18558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org