You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2026/03/28 10:43:08 UTC

[PR] Add SegmentPruner support for RestrictedDataSource policy filters (druid)

clintropolis opened a new pull request, #19228:
URL: https://github.com/apache/druid/pull/19228

   changes:
   * adds new `include` method to `SegmentPruner` for checking individual segments for whether or not to prune
   * adds default implementation of `prune` method which calls `include`
   * adds new `combine` method to `SegmentPruner` for merging pruners
   * adds new `CompositeSegmentPruner` for cases where pruners cannot be naturally combined
   * adds new `createSegmentPruner` method to `DataSource` and `Policy` so that they can participate in pruning
   * updates `ExecutionVertex` to combine the new datasource pruner with the pruner of the filter


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] feat: add SegmentPruner support for datasources/policies (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #19228:
URL: https://github.com/apache/druid/pull/19228#discussion_r3005401023


##########
processing/src/main/java/org/apache/druid/query/DataSource.java:
##########
@@ -105,6 +106,19 @@ public interface DataSource extends Cacheable
    */
   SegmentMapFunction createSegmentMapFunction(Query query);
 
+  /**
+   * Returns a {@link SegmentPruner} if this datasource embeds in any information which can be used to determine if a
+   * segment needs processed or not. Note that callers of this method will always only be processing segments for the
+   * datasource, so there is no need for a 'default' pruner that ensures the segment has the proper datasource. A return
+   * value of null indicates that no pruning can be performed from this datasource, though other sources of pruning,
+   * such as filters may still be used.
+   */
+  @Nullable
+  default SegmentPruner createSegmentPruner()

Review Comment:
   Could also implement this in `FilteredDataSource`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] feat: add SegmentPruner support for datasources/policies (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis merged PR #19228:
URL: https://github.com/apache/druid/pull/19228


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] feat: add SegmentPruner support for datasources/policies (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19228:
URL: https://github.com/apache/druid/pull/19228#discussion_r3007353432


##########
processing/src/main/java/org/apache/druid/query/DataSource.java:
##########
@@ -105,6 +106,19 @@ public interface DataSource extends Cacheable
    */
   SegmentMapFunction createSegmentMapFunction(Query query);
 
+  /**
+   * Returns a {@link SegmentPruner} if this datasource embeds in any information which can be used to determine if a
+   * segment needs processed or not. Note that callers of this method will always only be processing segments for the
+   * datasource, so there is no need for a 'default' pruner that ensures the segment has the proper datasource. A return
+   * value of null indicates that no pruning can be performed from this datasource, though other sources of pruning,
+   * such as filters may still be used.
+   */
+  @Nullable
+  default SegmentPruner createSegmentPruner()

Review Comment:
   yea, I thought about this, but ignored it for now since I think `FilteredDataSource`, and `UnnestDataSource` since it has a filter too, both need to be a bit more thoughtful in how they prune. I think they need to be combining with the pruner that is beneath them from the base, but maybe only in some cases or modifying it in others? Like I think for unnest we might want to like prune differently if the filter is on the unnest column, depending on whether unnest is on a mvd or an array, similar to what we do for unnest filter pushdown? I'm not certain if we have to do anything besides combine `FilteredDataSource`, I haven't fully thought about it yet, and didn't want to for now 😅 
   
   I'll look into improving this in a follow-up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org