You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/07/25 03:35:25 UTC
[GitHub] [doris] englefly opened a new pull request, #11162: [feature] add rule to push down predicate through aggregate
englefly opened a new pull request, #11162:
URL: https://github.com/apache/doris/pull/11162
# Proposed changes
add rule to push predicates down to aggregation node
1. add PushDownPredicatesThroughAggregation.java
2. add ut for PushDownPredicatesThroughAggregation
* For example:
* Logical plan tree:
* any_node
* |
* filter (a>0 and b>0)
* |
* group by(a, c)
* |
* scan
* transformed to:
* project
* |
* upper filter (b>0)
* |
* group by(a, c)
* |
* bottom filter (a>0)
* |
* scan
* Note:
* 'a>0' could be push down, because 'a' is in group by keys;
* but 'b>0' could not push down, because 'b' is not in group by keys.
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
4. Has unit tests been added: (Yes/No/No Need)
5. Has document been added or modified: (Yes/No/No Need)
6. Does it need to update dependencies: (Yes/No)
7. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
wangshuo128 commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928463589
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
Review Comment:
Oh, I see. I think we should find expressions that satisfy a one-to-one mapping and optimizer further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928458543
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
+ groupBySlots.add((Slot) groupByExpression);
+ }
+ }
+ List<Expression> pushDownPredicates = Lists.newArrayList();
+ List<Expression> filterPredicates = Lists.newArrayList();
+ ExpressionUtils.extractConjunct(filter.getPredicates()).forEach(conjunct -> {
+ Set<Slot> conjunctSlots = SlotExtractor.extractSlot(conjunct);
+ if (groupBySlots.containsAll(conjunctSlots)) {
+ pushDownPredicates.add(conjunct);
+ } else {
+ //TODO here
+ filterPredicates.add(conjunct);
+ }
+ });
+
+ return pushDownPredicate(filter, aggregate, pushDownPredicates, filterPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_AGGREGATION);
+ }
+
+ private Plan pushDownPredicate(LogicalFilter filter, LogicalAggregate aggregate,
+ List<Expression> pushDownPredicates, List<Expression> filterPredicates) {
+ Plan root;
+ if (filterPredicates.isEmpty()) {
+ //all predicates are pushed down, just exchange filter and aggregate
+ LogicalFilter bottomFilter = new LogicalFilter(ExpressionUtils.and(pushDownPredicates),
+ (Plan) aggregate.child(0));
+ root = aggregate.withChildren(Lists.newArrayList(bottomFilter));
+ } else {
+ LogicalFilter bottomFilter = new LogicalFilter(ExpressionUtils.and(pushDownPredicates),
Review Comment:
`pushDownPredicates` is not judged to be empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928457267
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
Review Comment:
It is may generate nonequivalent plans if we use `SlotExtractor`.
Consider a more comprehensive function on `a`. For example:
we have a function `f(x)`
`f(a) = 0, if a is even`
`f(a)=1, if a is odd`
the origin plan is : `filter(a>2) -->group by f(a)`
The result should be empty.
but if we push `a>2` down, the result changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11162: [feature] (Nereids) add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928578590
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
Review Comment:
@924060929 here is a example about your concern
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
wangshuo128 commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928455513
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
+ groupBySlots.add((Slot) groupByExpression);
+ }
+ }
+ List<Expression> pushDownPredicates = Lists.newArrayList();
+ List<Expression> filterPredicates = Lists.newArrayList();
+ ExpressionUtils.extractConjunct(filter.getPredicates()).forEach(conjunct -> {
+ Set<Slot> conjunctSlots = SlotExtractor.extractSlot(conjunct);
+ if (groupBySlots.containsAll(conjunctSlots)) {
+ pushDownPredicates.add(conjunct);
+ } else {
+ //TODO here
Review Comment:
Would you please elaborate the details to be done?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11162: [feature] (Nereids) add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928575300
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
+ groupBySlots.add((Slot) groupByExpression);
+ }
+ }
+ List<Expression> pushDownPredicates = Lists.newArrayList();
+ List<Expression> filterPredicates = Lists.newArrayList();
+ ExpressionUtils.extractConjunct(filter.getPredicates()).forEach(conjunct -> {
+ Set<Slot> conjunctSlots = SlotExtractor.extractSlot(conjunct);
+ if (groupBySlots.containsAll(conjunctSlots)) {
+ pushDownPredicates.add(conjunct);
+ } else {
+ //TODO here
+ filterPredicates.add(conjunct);
+ }
+ });
+
+ return pushDownPredicate(filter, aggregate, pushDownPredicates, filterPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_AGGREGATION);
+ }
+
+ private Plan pushDownPredicate(LogicalFilter filter, LogicalAggregate aggregate,
+ List<Expression> pushDownPredicates, List<Expression> filterPredicates) {
+ Plan root;
Review Comment:
@924060929 here we discussed your concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11162: [feature] (Nereids) add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #11162:
URL: https://github.com/apache/doris/pull/11162#issuecomment-1196273644
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11162: [feature] (Nereids) add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928579440
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
+ groupBySlots.add((Slot) groupByExpression);
+ }
+ }
+ List<Expression> pushDownPredicates = Lists.newArrayList();
+ List<Expression> filterPredicates = Lists.newArrayList();
+ ExpressionUtils.extractConjunct(filter.getPredicates()).forEach(conjunct -> {
+ Set<Slot> conjunctSlots = SlotExtractor.extractSlot(conjunct);
+ if (groupBySlots.containsAll(conjunctSlots)) {
+ pushDownPredicates.add(conjunct);
+ } else {
+ //TODO here
+ filterPredicates.add(conjunct);
+ }
+ });
+
+ return pushDownPredicate(filter, aggregate, pushDownPredicates, filterPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_AGGREGATION);
+ }
+
+ private Plan pushDownPredicate(LogicalFilter filter, LogicalAggregate aggregate,
+ List<Expression> pushDownPredicates, List<Expression> filterPredicates) {
+ Plan root;
Review Comment:
code was refactored, add `return` in if-else clause
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 merged pull request #11162: [feature] (Nereids) add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
924060929 merged PR #11162:
URL: https://github.com/apache/doris/pull/11162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
wangshuo128 commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928463993
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
+ groupBySlots.add((Slot) groupByExpression);
+ }
+ }
+ List<Expression> pushDownPredicates = Lists.newArrayList();
+ List<Expression> filterPredicates = Lists.newArrayList();
+ ExpressionUtils.extractConjunct(filter.getPredicates()).forEach(conjunct -> {
+ Set<Slot> conjunctSlots = SlotExtractor.extractSlot(conjunct);
+ if (groupBySlots.containsAll(conjunctSlots)) {
+ pushDownPredicates.add(conjunct);
+ } else {
+ //TODO here
+ filterPredicates.add(conjunct);
+ }
+ });
+
+ return pushDownPredicate(filter, aggregate, pushDownPredicates, filterPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_AGGREGATION);
+ }
+
+ private Plan pushDownPredicate(LogicalFilter filter, LogicalAggregate aggregate,
+ List<Expression> pushDownPredicates, List<Expression> filterPredicates) {
+ Plan root;
Review Comment:
Yeah, it's about code style. Any thoughts about this? @924060929 @morrySnow @EmmyMiao87?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
wangshuo128 commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928435108
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
Review Comment:
Do we need to handle more comprehensive cases, e.g., `group by a + 1`?
Maybe use `SlotExtractor` to gather all the input slot references of a group by expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928458145
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
+ groupBySlots.add((Slot) groupByExpression);
+ }
+ }
+ List<Expression> pushDownPredicates = Lists.newArrayList();
+ List<Expression> filterPredicates = Lists.newArrayList();
+ ExpressionUtils.extractConjunct(filter.getPredicates()).forEach(conjunct -> {
+ Set<Slot> conjunctSlots = SlotExtractor.extractSlot(conjunct);
+ if (groupBySlots.containsAll(conjunctSlots)) {
+ pushDownPredicates.add(conjunct);
+ } else {
+ //TODO here
+ filterPredicates.add(conjunct);
+ }
+ });
+
+ return pushDownPredicate(filter, aggregate, pushDownPredicates, filterPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_AGGREGATION);
+ }
+
+ private Plan pushDownPredicate(LogicalFilter filter, LogicalAggregate aggregate,
+ List<Expression> pushDownPredicates, List<Expression> filterPredicates) {
+ Plan root;
Review Comment:
If we keep only one return in a function, it would be more readable.
Anyway, this is more like a code format style. If our team has such rule, I will follow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
wangshuo128 commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928450692
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
+ groupBySlots.add((Slot) groupByExpression);
+ }
+ }
+ List<Expression> pushDownPredicates = Lists.newArrayList();
+ List<Expression> filterPredicates = Lists.newArrayList();
+ ExpressionUtils.extractConjunct(filter.getPredicates()).forEach(conjunct -> {
+ Set<Slot> conjunctSlots = SlotExtractor.extractSlot(conjunct);
+ if (groupBySlots.containsAll(conjunctSlots)) {
+ pushDownPredicates.add(conjunct);
+ } else {
+ //TODO here
+ filterPredicates.add(conjunct);
+ }
+ });
+
+ return pushDownPredicate(filter, aggregate, pushDownPredicates, filterPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_AGGREGATION);
+ }
+
+ private Plan pushDownPredicate(LogicalFilter filter, LogicalAggregate aggregate,
+ List<Expression> pushDownPredicates, List<Expression> filterPredicates) {
+ Plan root;
Review Comment:
nit: directly return the local result in the `if else` branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928470976
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
Review Comment:
But there is no such information in Expression class. Maybe, we could add them in future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11162: [feature] (Nereids) add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928577324
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,109 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
Review Comment:
this is not a correct push-down. Refer to another comments, please. I mentioned you in that discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
wangshuo128 commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928463589
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
Review Comment:
Oh, I see. I think we should find expressions that satisfy a one-to-one mapping and enhance the logic further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928471149
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
+ groupBySlots.add((Slot) groupByExpression);
+ }
+ }
+ List<Expression> pushDownPredicates = Lists.newArrayList();
+ List<Expression> filterPredicates = Lists.newArrayList();
+ ExpressionUtils.extractConjunct(filter.getPredicates()).forEach(conjunct -> {
+ Set<Slot> conjunctSlots = SlotExtractor.extractSlot(conjunct);
+ if (groupBySlots.containsAll(conjunctSlots)) {
+ pushDownPredicates.add(conjunct);
+ } else {
+ //TODO here
Review Comment:
My fault. It should be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11162: [feature] (Nereids) add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928513870
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,109 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
Review Comment:
the filter can push down through global aggregate and local aggregate correctly?
e.g.
```java
LogicalFilter(slotRef(#1) = 100)
|- LogicalAggregate(groupBy=[slotRef(#1)], output=[slotRef(#1), sum(slotRef(#2))])
|- LogicalAggregate(groupBy=[a + 1], output=[Alias(a+1 as #1), Alias(sum(value) as #2)]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11162: [feature] (Nereids) add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928513870
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,109 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
Review Comment:
the filter can push down through global aggregate and local aggregate correctly?
e.g.
```java
LogicalFilter(slotRef(#1) = 100)
|- LogicalAggregate(groupBy=[Alias(#1)], output=[slotRef(#1), sum(slotRef(#2))])
|- LogicalAggregate(groupBy=[a + 1], output=[Alias(a+1 as #1), Alias(sum(value) as #2)]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11162: [feature] add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r928499127
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
+ groupBySlots.add((Slot) groupByExpression);
+ }
+ }
+ List<Expression> pushDownPredicates = Lists.newArrayList();
+ List<Expression> filterPredicates = Lists.newArrayList();
+ ExpressionUtils.extractConjunct(filter.getPredicates()).forEach(conjunct -> {
+ Set<Slot> conjunctSlots = SlotExtractor.extractSlot(conjunct);
+ if (groupBySlots.containsAll(conjunctSlots)) {
+ pushDownPredicates.add(conjunct);
+ } else {
+ //TODO here
+ filterPredicates.add(conjunct);
+ }
+ });
+
+ return pushDownPredicate(filter, aggregate, pushDownPredicates, filterPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_AGGREGATION);
+ }
+
+ private Plan pushDownPredicate(LogicalFilter filter, LogicalAggregate aggregate,
+ List<Expression> pushDownPredicates, List<Expression> filterPredicates) {
+ Plan root;
+ if (filterPredicates.isEmpty()) {
+ //all predicates are pushed down, just exchange filter and aggregate
+ LogicalFilter bottomFilter = new LogicalFilter(ExpressionUtils.and(pushDownPredicates),
+ (Plan) aggregate.child(0));
+ root = aggregate.withChildren(Lists.newArrayList(bottomFilter));
+ } else {
+ LogicalFilter bottomFilter = new LogicalFilter(ExpressionUtils.and(pushDownPredicates),
Review Comment:
done: add branch for this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11162: [feature] (Nereids) add rule to push down predicate through aggregate
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #11162:
URL: https://github.com/apache/doris/pull/11162#discussion_r930601833
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughAggregation.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.visitor.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+/**
+ * Push the predicate in the LogicalFilter to the aggregate child.
+ * For example:
+ * Logical plan tree:
+ * any_node
+ * |
+ * filter (a>0 and b>0)
+ * |
+ * group by(a, c)
+ * |
+ * scan
+ * transformed to:
+ * project
+ * |
+ * upper filter (b>0)
+ * |
+ * group by(a, c)
+ * |
+ * bottom filter (a>0)
+ * |
+ * scan
+ * Note:
+ * 'a>0' could be push down, because 'a' is in group by keys;
+ * but 'b>0' could not push down, because 'b' is not in group by keys.
+ *
+ */
+
+public class PushPredicateThroughAggregation extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalFilter(logicalAggregate()).then(filter -> {
+ LogicalAggregate<GroupPlan> aggregate = filter.child();
+ Set<Slot> groupBySlots = new HashSet<>();
+ for (Expression groupByExpression : aggregate.getGroupByExpressionList()) {
+ if (groupByExpression instanceof Slot) {
Review Comment:
i do not understand this example. why could do filter(a > 2) after group by f(a), since a is neither key nor aggregate function output.
If u mean
```
Filter(f(a) > 2)
|--Aggregate([sum(b)], [f(a)])
```
then, i think the filter should could be pushed down aggregate node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org