You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2024/12/16 08:32:02 UTC

[PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Jackie-Jiang opened a new pull request, #14664:
URL: https://github.com/apache/pinot/pull/14664

   Introduce `is_enable_group_trim` as an agg option to enable group trim in the leaf stage.
   Group trim can be enabled when there is order by and limit.
   
   E.g.
   ```
   SELECT /*+ aggOptions(is_partitioned_by_group_by_keys='true', is_enable_group_trim='true') */ {tbl1}.num, COUNT(*), SUM({tbl1}.val), SUM({tbl1}.num), COUNT(DISTINCT {tbl1}.val) FROM {tbl1} WHERE {tbl1}.val >= 0 AND {tbl1}.name != 'a' GROUP BY {tbl1}.num ORDER BY COUNT(*) DESC LIMIT 1
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #14664:
URL: https://github.com/apache/pinot/pull/14664#discussion_r1899322104


##########
pinot-query-runtime/src/test/resources/queries/QueryHints.json:
##########
@@ -321,6 +321,10 @@
         "description": "aggregate with skip intermediate stage hint (via hint option is_partitioned_by_group_by_keys)",
         "sql": "SELECT /*+ aggOptions(is_partitioned_by_group_by_keys='true') */ {tbl1}.num, COUNT(*), SUM({tbl1}.val), SUM({tbl1}.num), COUNT(DISTINCT {tbl1}.val) FROM {tbl1} WHERE {tbl1}.val >= 0 AND {tbl1}.name != 'a' GROUP BY {tbl1}.num"
       },

Review Comment:
   I can probably make one, but that will cause wrong (and inconsistent) result, so not sure how to test it. I've manually tested it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #14664:
URL: https://github.com/apache/pinot/pull/14664#discussion_r1899321259


##########
pinot-query-planner/src/test/resources/queries/GroupByPlans.json:
##########
@@ -249,6 +249,39 @@
           "\n              LogicalTableScan(table=[[default, a]])",
           "\n"
         ]
+      },

Review Comment:
   Leaf stage has both segment trim and server trim. Will add some javadoc about the behavior, and we should consider adding them also as v2 hint. Right now the behavior relies on the v1 query options.
   
   Agree on automating it when the result is guaranteed to be accurate. This will also require some v1 engine change, so will do it separately.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "bziobrowski (via GitHub)" <gi...@apache.org>.
bziobrowski commented on code in PR #14664:
URL: https://github.com/apache/pinot/pull/14664#discussion_r1890464726


##########
pinot-query-planner/src/test/resources/queries/GroupByPlans.json:
##########
@@ -249,6 +249,39 @@
           "\n              LogicalTableScan(table=[[default, a]])",
           "\n"
         ]
+      },

Review Comment:
   Is this PR meant to enable trimming within (minSegmentTrim) and across segments (minServerTrim) ?
   If so - It'd be good to mention that :
   - the former is disabled by default  and requires using query option (or cluster setting)
   - the latter is enabled by default but probably haven't been applied so far
   
   I reckon It'd be good to also test this hint has on a query without  order by and/or limit clause.
   
   Apart from manual hinting I think we could propagate limit and order by details to leaf plans and enable both types of trimming if: 
   - order by is based on group by key(s) only (not aggregates)
   - there is no HAVING clause



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "bziobrowski (via GitHub)" <gi...@apache.org>.
bziobrowski commented on code in PR #14664:
URL: https://github.com/apache/pinot/pull/14664#discussion_r1899462490


##########
pinot-query-runtime/src/test/resources/queries/QueryHints.json:
##########
@@ -321,6 +321,10 @@
         "description": "aggregate with skip intermediate stage hint (via hint option is_partitioned_by_group_by_keys)",
         "sql": "SELECT /*+ aggOptions(is_partitioned_by_group_by_keys='true') */ {tbl1}.num, COUNT(*), SUM({tbl1}.val), SUM({tbl1}.num), COUNT(DISTINCT {tbl1}.val) FROM {tbl1} WHERE {tbl1}.val >= 0 AND {tbl1}.name != 'a' GROUP BY {tbl1}.num"
       },

Review Comment:
   I already added some tests in https://github.com/apache/pinot/pull/14727.
   It was tricky because in many cases trimming makes result unstable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_push_down_limit agg option [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #14664:
URL: https://github.com/apache/pinot/pull/14664#discussion_r1899317112


##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRule.java:
##########
@@ -83,48 +85,148 @@
  *   - COUNT(*)__LEAF produces TUPLE[ SUM(1), GROUP_BY_KEY ]
  *   - COUNT(*)__FINAL produces TUPLE[ SUM(COUNT(*)__LEAF), GROUP_BY_KEY ]
  */
-public class PinotAggregateExchangeNodeInsertRule extends RelOptRule {
-  public static final PinotAggregateExchangeNodeInsertRule INSTANCE =
-      new PinotAggregateExchangeNodeInsertRule(PinotRuleUtils.PINOT_REL_FACTORY);
-
-  public PinotAggregateExchangeNodeInsertRule(RelBuilderFactory factory) {
-    // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
-    //       PinotLogicalAggregate, and the rule won't be applied again.
-    super(operand(LogicalAggregate.class, any()), factory, null);
+public class PinotAggregateExchangeNodeInsertRule {
+
+  public static class SortProjectAggregate extends RelOptRule {
+    public static final SortProjectAggregate INSTANCE = new SortProjectAggregate(PinotRuleUtils.PINOT_REL_FACTORY);
+
+    private SortProjectAggregate(RelBuilderFactory factory) {
+      // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
+      //       PinotLogicalAggregate, and the rule won't be applied again.
+      super(operand(Sort.class, operand(Project.class, operand(LogicalAggregate.class, any()))), factory, null);
+    }
+
+    @Override
+    public void onMatch(RelOptRuleCall call) {
+      // Apply this rule for group-by queries with enable group trim hint.
+      LogicalAggregate aggRel = call.rel(2);
+      if (aggRel.getGroupSet().isEmpty()) {
+        return;
+      }
+      Map<String, String> hintOptions =
+          PinotHintStrategyTable.getHintOptions(aggRel.getHints(), PinotHintOptions.AGGREGATE_HINT_OPTIONS);
+      if (hintOptions == null || !Boolean.parseBoolean(
+          hintOptions.get(PinotHintOptions.AggregateOptions.ENABLE_GROUP_TRIM))) {
+        return;
+      }
+
+      Sort sortRel = call.rel(0);
+      Project projectRel = call.rel(1);
+      List<RexNode> projects = projectRel.getProjects();
+      List<RelFieldCollation> collations = sortRel.getCollation().getFieldCollations();
+      if (collations.isEmpty()) {
+        // Cannot enable group trim without sort key.

Review Comment:
   Limit can be pushed down with or without sorting. Initially I decided to not allow pushing down limit without sorting because it can cause wrong result for aggregate group-by, but realized this can prevent distinct limit to be pushed down. Since this is not enabled by default (requires hint), I'll change it to not check collation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #14664:
URL: https://github.com/apache/pinot/pull/14664#issuecomment-2550563693

   cc @bziobrowski 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "bziobrowski (via GitHub)" <gi...@apache.org>.
bziobrowski commented on code in PR #14664:
URL: https://github.com/apache/pinot/pull/14664#discussion_r1890464726


##########
pinot-query-planner/src/test/resources/queries/GroupByPlans.json:
##########
@@ -249,6 +249,39 @@
           "\n              LogicalTableScan(table=[[default, a]])",
           "\n"
         ]
+      },

Review Comment:
   Is this PR meant to enable trimming within (minSegmentTrim) and across segments (minServerTrim) ?
   If so - It'd be good to mention that :
   - the former is disabled by default  and requires using query option (or cluster setting)
   - the latter is enabled by default but probably haven't been applied so far
   
   Apart from manual hinting I think we could propagate limit and order by details to leaf plans and enable both types of trimming if: 
   - order by is based on group by key(s) only (not aggregates)
   - there is no HAVING clause



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "bziobrowski (via GitHub)" <gi...@apache.org>.
bziobrowski commented on code in PR #14664:
URL: https://github.com/apache/pinot/pull/14664#discussion_r1890471070


##########
pinot-query-runtime/src/test/resources/queries/QueryHints.json:
##########
@@ -321,6 +321,10 @@
         "description": "aggregate with skip intermediate stage hint (via hint option is_partitioned_by_group_by_keys)",
         "sql": "SELECT /*+ aggOptions(is_partitioned_by_group_by_keys='true') */ {tbl1}.num, COUNT(*), SUM({tbl1}.val), SUM({tbl1}.num), COUNT(DISTINCT {tbl1}.val) FROM {tbl1} WHERE {tbl1}.val >= 0 AND {tbl1}.name != 'a' GROUP BY {tbl1}.num"
       },

Review Comment:
   Is there a test asserting that trimming affects results ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #14664:
URL: https://github.com/apache/pinot/pull/14664


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #14664:
URL: https://github.com/apache/pinot/pull/14664#discussion_r1899317112


##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRule.java:
##########
@@ -83,48 +85,148 @@
  *   - COUNT(*)__LEAF produces TUPLE[ SUM(1), GROUP_BY_KEY ]
  *   - COUNT(*)__FINAL produces TUPLE[ SUM(COUNT(*)__LEAF), GROUP_BY_KEY ]
  */
-public class PinotAggregateExchangeNodeInsertRule extends RelOptRule {
-  public static final PinotAggregateExchangeNodeInsertRule INSTANCE =
-      new PinotAggregateExchangeNodeInsertRule(PinotRuleUtils.PINOT_REL_FACTORY);
-
-  public PinotAggregateExchangeNodeInsertRule(RelBuilderFactory factory) {
-    // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
-    //       PinotLogicalAggregate, and the rule won't be applied again.
-    super(operand(LogicalAggregate.class, any()), factory, null);
+public class PinotAggregateExchangeNodeInsertRule {
+
+  public static class SortProjectAggregate extends RelOptRule {
+    public static final SortProjectAggregate INSTANCE = new SortProjectAggregate(PinotRuleUtils.PINOT_REL_FACTORY);
+
+    private SortProjectAggregate(RelBuilderFactory factory) {
+      // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
+      //       PinotLogicalAggregate, and the rule won't be applied again.
+      super(operand(Sort.class, operand(Project.class, operand(LogicalAggregate.class, any()))), factory, null);
+    }
+
+    @Override
+    public void onMatch(RelOptRuleCall call) {
+      // Apply this rule for group-by queries with enable group trim hint.
+      LogicalAggregate aggRel = call.rel(2);
+      if (aggRel.getGroupSet().isEmpty()) {
+        return;
+      }
+      Map<String, String> hintOptions =
+          PinotHintStrategyTable.getHintOptions(aggRel.getHints(), PinotHintOptions.AGGREGATE_HINT_OPTIONS);
+      if (hintOptions == null || !Boolean.parseBoolean(
+          hintOptions.get(PinotHintOptions.AggregateOptions.ENABLE_GROUP_TRIM))) {
+        return;
+      }
+
+      Sort sortRel = call.rel(0);
+      Project projectRel = call.rel(1);
+      List<RexNode> projects = projectRel.getProjects();
+      List<RelFieldCollation> collations = sortRel.getCollation().getFieldCollations();
+      if (collations.isEmpty()) {
+        // Cannot enable group trim without sort key.

Review Comment:
   Limit can be pushed down with or without sorting. Initially I decided to not allow pushing down limit without sorting because it can cause wrong result for aggregate group-by, but realized this can prevent distinct limit to be pushed down. Since this is not enabled by default (requires hint), I'll change it to not check collation. I'm thinking also rename the hint to `is_push_down_limit`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #14664:
URL: https://github.com/apache/pinot/pull/14664#discussion_r1899319481


##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRule.java:
##########
@@ -83,48 +85,148 @@
  *   - COUNT(*)__LEAF produces TUPLE[ SUM(1), GROUP_BY_KEY ]
  *   - COUNT(*)__FINAL produces TUPLE[ SUM(COUNT(*)__LEAF), GROUP_BY_KEY ]
  */
-public class PinotAggregateExchangeNodeInsertRule extends RelOptRule {
-  public static final PinotAggregateExchangeNodeInsertRule INSTANCE =
-      new PinotAggregateExchangeNodeInsertRule(PinotRuleUtils.PINOT_REL_FACTORY);
-
-  public PinotAggregateExchangeNodeInsertRule(RelBuilderFactory factory) {
-    // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
-    //       PinotLogicalAggregate, and the rule won't be applied again.
-    super(operand(LogicalAggregate.class, any()), factory, null);
+public class PinotAggregateExchangeNodeInsertRule {
+
+  public static class SortProjectAggregate extends RelOptRule {
+    public static final SortProjectAggregate INSTANCE = new SortProjectAggregate(PinotRuleUtils.PINOT_REL_FACTORY);
+
+    private SortProjectAggregate(RelBuilderFactory factory) {
+      // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
+      //       PinotLogicalAggregate, and the rule won't be applied again.
+      super(operand(Sort.class, operand(Project.class, operand(LogicalAggregate.class, any()))), factory, null);
+    }
+
+    @Override
+    public void onMatch(RelOptRuleCall call) {

Review Comment:
   Some Calcite rules also rely on return in `onMatch()` if the check is not trivial to reduce overhead, e.g. `ProjectToSemiJoinRule`. I'm not sure if it is a good idea to rely on which method is being invoked. IMO a safer way to detect which rules are applied is to check the actual rewrite of RelNode



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14664:
URL: https://github.com/apache/pinot/pull/14664#discussion_r1889890059


##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRule.java:
##########
@@ -83,48 +85,148 @@
  *   - COUNT(*)__LEAF produces TUPLE[ SUM(1), GROUP_BY_KEY ]
  *   - COUNT(*)__FINAL produces TUPLE[ SUM(COUNT(*)__LEAF), GROUP_BY_KEY ]
  */
-public class PinotAggregateExchangeNodeInsertRule extends RelOptRule {
-  public static final PinotAggregateExchangeNodeInsertRule INSTANCE =
-      new PinotAggregateExchangeNodeInsertRule(PinotRuleUtils.PINOT_REL_FACTORY);
-
-  public PinotAggregateExchangeNodeInsertRule(RelBuilderFactory factory) {
-    // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
-    //       PinotLogicalAggregate, and the rule won't be applied again.
-    super(operand(LogicalAggregate.class, any()), factory, null);
+public class PinotAggregateExchangeNodeInsertRule {
+
+  public static class SortProjectAggregate extends RelOptRule {
+    public static final SortProjectAggregate INSTANCE = new SortProjectAggregate(PinotRuleUtils.PINOT_REL_FACTORY);
+
+    private SortProjectAggregate(RelBuilderFactory factory) {
+      // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
+      //       PinotLogicalAggregate, and the rule won't be applied again.
+      super(operand(Sort.class, operand(Project.class, operand(LogicalAggregate.class, any()))), factory, null);
+    }
+
+    @Override
+    public void onMatch(RelOptRuleCall call) {

Review Comment:
   nit:
   
   Calcite we have `onMatch` and `matches` methods. This method seems to be doing both. I think it would be better to override `matches` to do not apply the rule in all the cases that are filtered out here.
   
   For the final query is the same and we would need to repeat some code, but the advantage of using _matches_ is that if try to debug which rules apply (using MarkerFilter, as explained in https://youtu.be/_phzRNCWJfw?si=hH9ukXAS2Iml11nq&t=331).
   
   I've just created https://github.com/apache/pinot/pull/14680 to do not forget how to enable these logs.



##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRule.java:
##########
@@ -83,48 +85,148 @@
  *   - COUNT(*)__LEAF produces TUPLE[ SUM(1), GROUP_BY_KEY ]
  *   - COUNT(*)__FINAL produces TUPLE[ SUM(COUNT(*)__LEAF), GROUP_BY_KEY ]
  */
-public class PinotAggregateExchangeNodeInsertRule extends RelOptRule {
-  public static final PinotAggregateExchangeNodeInsertRule INSTANCE =
-      new PinotAggregateExchangeNodeInsertRule(PinotRuleUtils.PINOT_REL_FACTORY);
-
-  public PinotAggregateExchangeNodeInsertRule(RelBuilderFactory factory) {
-    // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
-    //       PinotLogicalAggregate, and the rule won't be applied again.
-    super(operand(LogicalAggregate.class, any()), factory, null);
+public class PinotAggregateExchangeNodeInsertRule {
+
+  public static class SortProjectAggregate extends RelOptRule {
+    public static final SortProjectAggregate INSTANCE = new SortProjectAggregate(PinotRuleUtils.PINOT_REL_FACTORY);
+
+    private SortProjectAggregate(RelBuilderFactory factory) {
+      // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
+      //       PinotLogicalAggregate, and the rule won't be applied again.
+      super(operand(Sort.class, operand(Project.class, operand(LogicalAggregate.class, any()))), factory, null);
+    }
+
+    @Override
+    public void onMatch(RelOptRuleCall call) {
+      // Apply this rule for group-by queries with enable group trim hint.
+      LogicalAggregate aggRel = call.rel(2);
+      if (aggRel.getGroupSet().isEmpty()) {
+        return;
+      }
+      Map<String, String> hintOptions =
+          PinotHintStrategyTable.getHintOptions(aggRel.getHints(), PinotHintOptions.AGGREGATE_HINT_OPTIONS);
+      if (hintOptions == null || !Boolean.parseBoolean(
+          hintOptions.get(PinotHintOptions.AggregateOptions.ENABLE_GROUP_TRIM))) {
+        return;
+      }
+
+      Sort sortRel = call.rel(0);
+      Project projectRel = call.rel(1);
+      List<RexNode> projects = projectRel.getProjects();
+      List<RelFieldCollation> collations = sortRel.getCollation().getFieldCollations();
+      if (collations.isEmpty()) {
+        // Cannot enable group trim without sort key.

Review Comment:
   I understand this is required for [within segment](https://docs.pinot.apache.org/users/user-guide-query/query-syntax/grouping-algorithm#within-segment) trimming right? Is it an actual requirement for cross segment trimming?
   
   Anyway, I'm not sure if we should make this kind of decisions here. Couldn't we populate the fields and let the actual operator decide whether an optimization can be applied or not?
   
   More specifically: Don't you think it may be interesting to keep the limit even if we don't have collations?



##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRule.java:
##########
@@ -83,48 +85,148 @@
  *   - COUNT(*)__LEAF produces TUPLE[ SUM(1), GROUP_BY_KEY ]
  *   - COUNT(*)__FINAL produces TUPLE[ SUM(COUNT(*)__LEAF), GROUP_BY_KEY ]
  */
-public class PinotAggregateExchangeNodeInsertRule extends RelOptRule {
-  public static final PinotAggregateExchangeNodeInsertRule INSTANCE =
-      new PinotAggregateExchangeNodeInsertRule(PinotRuleUtils.PINOT_REL_FACTORY);
-
-  public PinotAggregateExchangeNodeInsertRule(RelBuilderFactory factory) {
-    // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
-    //       PinotLogicalAggregate, and the rule won't be applied again.
-    super(operand(LogicalAggregate.class, any()), factory, null);
+public class PinotAggregateExchangeNodeInsertRule {
+
+  public static class SortProjectAggregate extends RelOptRule {

Review Comment:
   Although the javadoc in the main class applies to all rules, it would be cool to add a javadoc for each rule to explain what they do (and maybe add an example)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "bziobrowski (via GitHub)" <gi...@apache.org>.
bziobrowski commented on code in PR #14664:
URL: https://github.com/apache/pinot/pull/14664#discussion_r1890625755


##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotAggregateExchangeNodeInsertRule.java:
##########
@@ -83,48 +85,148 @@
  *   - COUNT(*)__LEAF produces TUPLE[ SUM(1), GROUP_BY_KEY ]
  *   - COUNT(*)__FINAL produces TUPLE[ SUM(COUNT(*)__LEAF), GROUP_BY_KEY ]
  */
-public class PinotAggregateExchangeNodeInsertRule extends RelOptRule {
-  public static final PinotAggregateExchangeNodeInsertRule INSTANCE =
-      new PinotAggregateExchangeNodeInsertRule(PinotRuleUtils.PINOT_REL_FACTORY);
-
-  public PinotAggregateExchangeNodeInsertRule(RelBuilderFactory factory) {
-    // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
-    //       PinotLogicalAggregate, and the rule won't be applied again.
-    super(operand(LogicalAggregate.class, any()), factory, null);
+public class PinotAggregateExchangeNodeInsertRule {
+
+  public static class SortProjectAggregate extends RelOptRule {
+    public static final SortProjectAggregate INSTANCE = new SortProjectAggregate(PinotRuleUtils.PINOT_REL_FACTORY);
+
+    private SortProjectAggregate(RelBuilderFactory factory) {
+      // NOTE: Explicitly match for LogicalAggregate because after applying the rule, LogicalAggregate is replaced with
+      //       PinotLogicalAggregate, and the rule won't be applied again.
+      super(operand(Sort.class, operand(Project.class, operand(LogicalAggregate.class, any()))), factory, null);
+    }
+
+    @Override
+    public void onMatch(RelOptRuleCall call) {
+      // Apply this rule for group-by queries with enable group trim hint.
+      LogicalAggregate aggRel = call.rel(2);
+      if (aggRel.getGroupSet().isEmpty()) {
+        return;
+      }
+      Map<String, String> hintOptions =
+          PinotHintStrategyTable.getHintOptions(aggRel.getHints(), PinotHintOptions.AGGREGATE_HINT_OPTIONS);
+      if (hintOptions == null || !Boolean.parseBoolean(
+          hintOptions.get(PinotHintOptions.AggregateOptions.ENABLE_GROUP_TRIM))) {
+        return;
+      }
+
+      Sort sortRel = call.rel(0);
+      Project projectRel = call.rel(1);
+      List<RexNode> projects = projectRel.getProjects();
+      List<RelFieldCollation> collations = sortRel.getCollation().getFieldCollations();
+      if (collations.isEmpty()) {
+        // Cannot enable group trim without sort key.

Review Comment:
   Having non-empty collation is required for trimming within segment and cross-segment but it could be beneficial to propagate limit even if collation is missing.
   That is because when limit is present (!= Integer.MAX_VALUE) combine operator might limit the number of group by keys in indexed table.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Multi-stage] Support is_enable_group_trim agg option [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #14664:
URL: https://github.com/apache/pinot/pull/14664#issuecomment-2545010873

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14664?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`aa3bf27`)](https://app.codecov.io/gh/apache/pinot/commit/aa3bf2773d22a2896a11a51ff9af68dfcf7e2d30?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 1472 commits behind head on master.
   
   > :exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (aa3bf27). Click for more details.
   > 
   > <details><summary>HEAD has 48 uploads less than BASE</summary>
   >
   >| Flag | BASE (59551e4) | HEAD (aa3bf27) |
   >|------|------|------|
   >|integration|7|2|
   >|integration2|3|2|
   >|temurin|12|2|
   >|java-21|7|2|
   >|skip-bytebuffers-true|3|1|
   >|skip-bytebuffers-false|7|1|
   >|unittests|5|0|
   >|unittests1|2|0|
   >|java-11|5|0|
   >|unittests2|3|0|
   >|integration1|2|0|
   >|custom-integration1|2|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #14664       +/-   ##
   =============================================
   - Coverage     61.75%    0.00%   -61.76%     
   =============================================
     Files          2436        3     -2433     
     Lines        133233        6   -133227     
     Branches      20636        0    -20636     
   =============================================
   - Hits          82274        0    -82274     
   + Misses        44911        6    -44905     
   + Partials       6048        0     -6048     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-0.01%)` | :arrow_down: |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.63%)` | :arrow_down: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.75%)` | :arrow_down: |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-27.73%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-61.76%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/14664/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/14664?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org