You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2026/06/06 09:18:52 UTC

[PR] fix: empty loads for asymmetric cluster-group partial-load matchers (druid)

clintropolis opened a new pull request, #19565:
URL: https://github.com/apache/druid/pull/19565

   ### Description
   This PR fixes a flaw with `ClusterGroupPartialLoadMatcher.match()` when it asymmetrically matches clustered segments within a core-partition group, resulting in only some of the core-partition segments in the group being loaded and being 'incomplete' in the timeline.
   
   The fix is to allow `PartialLoadMatcher` to perform 'empty' loads, where if the rule matcher is processing a clustered segment but none of the cluster groups match. Coupled with this, `RunRules` guards against totally empty groups by using a new `EmptyDeferringHandler` wrapper `SegmentActionHandler` that buffers these empty requests and 'flushes' them at shard-group borders, only performing empty loads when some part of a core partition group was a non-empty match.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: empty loads for asymmetric cluster-group partial-load matchers (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #19565:
URL: https://github.com/apache/druid/pull/19565#discussion_r3369495011


##########
server/src/main/java/org/apache/druid/server/coordinator/rules/ClusterGroupPartialLoadMatcher.java:
##########
@@ -59,10 +78,13 @@ public MatchResult match(DataSegment segment, Map<String, Object> baseLoadSpec)
       return null;
     }
     final List<Integer> resolved = resolveClusterGroupIndices(segment);
-    if (resolved.isEmpty()) {
+    final ShardSpec shardSpec = segment.getShardSpec();
+    if (resolved.isEmpty() && shardSpec.getPartitionNum() >= shardSpec.getNumCorePartitions()) {
+      // No patterns match and this segment isn't part of a core partition group, so no need for an empty load. Fall
+      // through to the cannot-match handling.
       return null;
     }
-    final String fingerprint = computeFingerprint(resolved);
+    final String fingerprint = resolved.isEmpty() ? EMPTY_LOAD_FINGERPRINT : computeFingerprint(resolved);

Review Comment:
   [P1] Empty matches bypass FALL_THROUGH for fully unmatched groups
   
   Returning a non-null empty MatchResult for every core segment means PartialLoadRule.appliesTo returns true even when the rule is configured with onCannotMatch=FALL_THROUGH. If a core shard group has clusterGroups but none of the segments match the configured patterns, RunRules stops on this partial rule for every segment, then EmptyDeferringHandler discards all buffered empties because no positive was seen. Later fallback rules never run, so segments that previously fell through to a full-load rule can be left with no assignment. The group-level discard path needs to preserve the configured cannot-match behavior.



##########
processing/src/main/java/org/apache/druid/segment/loading/PartialClusterGroupLoadSpec.java:
##########
@@ -72,10 +71,9 @@ public PartialClusterGroupLoadSpec(
   )
   {
     super(delegate, fingerprint, jsonMapper);
-    Preconditions.checkArgument(
-        !CollectionUtils.isNullOrEmpty(clusterGroupIndices),
-        "clusterGroupIndices must not be null or empty"
-    );
+    // An empty index list is used when a cluster-group matcher applies to a (clustered) segment but no configured
+    // pattern matches its cluster groups. The historical-side partial loader honors this by loading nothing.
+    Preconditions.checkNotNull(clusterGroupIndices, "clusterGroupIndices");

Review Comment:
   [P2] Empty loads still delegate to a full segment load
   
   The constructor now accepts an empty clusterGroupIndices list for the empty-load sentinel, but PartialClusterGroupLoadSpec still inherits PartialLoadSpec.loadSegment(), which delegates directly to the inner load spec. A historical receiving this supposedly empty request will download the full segment and announce a matching full-fallback profile, so sparse matches can full-load every empty sibling instead of loading nothing. This needs an explicit empty-load implementation or another guard before emitting empty cluster-group load specs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: empty loads for asymmetric cluster-group partial-load matchers (druid)

Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe merged PR #19565:
URL: https://github.com/apache/druid/pull/19565


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: empty loads for asymmetric cluster-group partial-load matchers (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19565:
URL: https://github.com/apache/druid/pull/19565#discussion_r3382811162


##########
server/src/main/java/org/apache/druid/server/coordinator/rules/ClusterGroupPartialLoadMatcher.java:
##########
@@ -59,10 +78,13 @@ public MatchResult match(DataSegment segment, Map<String, Object> baseLoadSpec)
       return null;
     }
     final List<Integer> resolved = resolveClusterGroupIndices(segment);
-    if (resolved.isEmpty()) {
+    final ShardSpec shardSpec = segment.getShardSpec();
+    if (resolved.isEmpty() && shardSpec.getPartitionNum() >= shardSpec.getNumCorePartitions()) {
+      // No patterns match and this segment isn't part of a core partition group, so no need for an empty load. Fall
+      // through to the cannot-match handling.
       return null;
     }
-    final String fingerprint = computeFingerprint(resolved);
+    final String fingerprint = resolved.isEmpty() ? EMPTY_LOAD_FINGERPRINT : computeFingerprint(resolved);

Review Comment:
   ah, so this is intentional behavior to not fall-through, however this comment made me realize that it was a bit too aggressive with the empty matches since it was applying to _any_ clustered segments.
   
   I only intended for empty matches to occur when the matcher is 'compatible' with the segment clustering, e.g. there is some overlap between what clustering columns the matcher is looking at and what columns the segment is actually clustered on so that we don't incorrect do an empty load for segments that are clustered completely differently from what the matcher is looking for.
   
   I've updated the check to ensure that empty matches only occur when things are compatible, which was the intended behavior



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: empty loads for asymmetric cluster-group partial-load matchers (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19565:
URL: https://github.com/apache/druid/pull/19565#discussion_r3385442405


##########
server/src/main/java/org/apache/druid/server/coordinator/rules/ClusterGroupPartialLoadMatcher.java:
##########
@@ -59,10 +78,13 @@ public MatchResult match(DataSegment segment, Map<String, Object> baseLoadSpec)
       return null;
     }
     final List<Integer> resolved = resolveClusterGroupIndices(segment);
-    if (resolved.isEmpty()) {
+    final ShardSpec shardSpec = segment.getShardSpec();
+    if (resolved.isEmpty() && shardSpec.getPartitionNum() >= shardSpec.getNumCorePartitions()) {
+      // No patterns match and this segment isn't part of a core partition group, so no need for an empty load. Fall
+      // through to the cannot-match handling.
       return null;
     }
-    final String fingerprint = computeFingerprint(resolved);
+    final String fingerprint = resolved.isEmpty() ? EMPTY_LOAD_FINGERPRINT : computeFingerprint(resolved);

Review Comment:
   actually still a problem here i think, trying to figure out the best thing to do



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: empty loads for asymmetric cluster-group partial-load matchers (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19565:
URL: https://github.com/apache/druid/pull/19565#discussion_r3382822045


##########
processing/src/main/java/org/apache/druid/segment/loading/PartialClusterGroupLoadSpec.java:
##########
@@ -72,10 +71,9 @@ public PartialClusterGroupLoadSpec(
   )
   {
     super(delegate, fingerprint, jsonMapper);
-    Preconditions.checkArgument(
-        !CollectionUtils.isNullOrEmpty(clusterGroupIndices),
-        "clusterGroupIndices must not be null or empty"
-    );
+    // An empty index list is used when a cluster-group matcher applies to a (clustered) segment but no configured
+    // pattern matches its cluster groups. The historical-side partial loader honors this by loading nothing.
+    Preconditions.checkNotNull(clusterGroupIndices, "clusterGroupIndices");

Review Comment:
   yea, this will be the case until partial load stuff is fully merged



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: empty loads for asymmetric cluster-group partial-load matchers (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19565:
URL: https://github.com/apache/druid/pull/19565#discussion_r3385442405


##########
server/src/main/java/org/apache/druid/server/coordinator/rules/ClusterGroupPartialLoadMatcher.java:
##########
@@ -59,10 +78,13 @@ public MatchResult match(DataSegment segment, Map<String, Object> baseLoadSpec)
       return null;
     }
     final List<Integer> resolved = resolveClusterGroupIndices(segment);
-    if (resolved.isEmpty()) {
+    final ShardSpec shardSpec = segment.getShardSpec();
+    if (resolved.isEmpty() && shardSpec.getPartitionNum() >= shardSpec.getNumCorePartitions()) {
+      // No patterns match and this segment isn't part of a core partition group, so no need for an empty load. Fall
+      // through to the cannot-match handling.
       return null;
     }
-    final String fingerprint = computeFingerprint(resolved);
+    final String fingerprint = resolved.isEmpty() ? EMPTY_LOAD_FINGERPRINT : computeFingerprint(resolved);

Review Comment:
   actually still a problem here i think, trying to figure out the best thing to do



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org