You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/06/28 02:53:56 UTC
[GitHub] [doris] qzsee opened a new pull request, #10462: [Enhancement](Nereids) push down predicate through join
qzsee opened a new pull request, #10462:
URL: https://github.com/apache/doris/pull/10462
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r909153447
##########
fe/fe-core/src/test/java/org/apache/doris/nereids/util/ExpressionUtilsTest.java:
##########
@@ -0,0 +1,84 @@
+package org.apache.doris.nereids.util;
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] EmmyMiao87 commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r909324956
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
Review Comment:
Can't find where this rule is registered
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_JOIN);
+ }
+
+ private Plan pushDownPredicate(LogicalBinaryPlan<LogicalJoin, GroupPlan, GroupPlan> joinPlan,
+ List<Expression> joinConditions, List<Expression> leftPredicates, List<Expression> rightPredicates) {
+
+ Expression left = ExpressionUtils.add(leftPredicates);
+ Expression right = ExpressionUtils.add(rightPredicates);
+ ExpressionRuleExecutor exprRewriter = new ExpressionRuleExecutor();
Review Comment:
Whether the rules related to expressions should be **split into two types** in the future.
One is the normalization rule on the original query. For example, ```BetweenToCompoundRule```, ```NormalizeExpressionRule``` etc.
One is optimization rules. Such as common factor extraction, deduplication etc.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_JOIN);
+ }
+
+ private Plan pushDownPredicate(LogicalBinaryPlan<LogicalJoin, GroupPlan, GroupPlan> joinPlan,
+ List<Expression> joinConditions, List<Expression> leftPredicates, List<Expression> rightPredicates) {
+
+ Expression left = ExpressionUtils.add(leftPredicates);
+ Expression right = ExpressionUtils.add(rightPredicates);
+ ExpressionRuleExecutor exprRewriter = new ExpressionRuleExecutor();
+ Plan leftPlan = joinPlan.left();
Review Comment:
Predicate pushdown should theoretically trigger expression rewriting rules again. But It actually only needs to call the second type rules.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
Review Comment:
When pushing down the predicate of filter below join, you need to consider the type of join.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
Review Comment:
Please include a clear real-world example in the comments.
Including the original query of the query, the original plan tree of the query, and the transformed new plan tree.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
Review Comment:
The fusion of join conditions cannot simply be extracted directly from the where and on clauses.
For example, the where predicate of outer join cannot be directly used as the condition of on. If it does, the result will be wrong.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
Review Comment:
The predicate pushdown rules on the on clause are completely different from the predicate pushdown rules in the where clause.
What you're providing here is predicate pushdown in the where clause, so I think your comment is wrong.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
Review Comment:
There are actually three predicate pushdowns related to join
1. The filter conditions are pushed up to join
2. The conditions of filter are pushed directly to join
3. The join condition is pushed to the following
where 1 is the basis of 2,3. Your pr only implements 1 and 2.
According to your implementation, the pushdown of these three predicates should be in one Rule rather than split into three Rules. I'm fine with that.
But you need to clearly represent these three different stages in the code (such as separate function processing).
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
Review Comment:
```suggestion
* todo: Now, only support eq condition in where clause, support other condition later
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908115394
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.util;
+
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+
+import com.google.common.collect.Lists;
+
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Expression rewrite helper class.
+ */
+public class ExpressionUtils {
+
+ public static final Literal TRUE_LITERAL = new Literal(true);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r911556011
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
Review Comment:
This version deals with inner join first, and other cases will be added later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r911556580
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
Review Comment:
This version deals with inner join first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r907997591
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/GroupExpressionMatching.java:
##########
@@ -133,6 +133,12 @@ private void assembleAllCombinationPlanTree(Plan root, Pattern<Plan, Plan> rootP
int[] childrenPlanIndex = new int[childrenPlans.size()];
int offset = 0;
+ for (List<Plan> plan : childrenPlans) {
Review Comment:
maybe we need to do this at L103
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/Job.java:
##########
@@ -57,7 +58,7 @@ public RuleSet getRuleSet() {
public List<Rule<NODE_TYPE>> getValidRules(GroupExpression groupExpression,
List<Rule<NODE_TYPE>> candidateRules) {
return candidateRules.stream()
- .filter(rule -> rule.getPattern().matchOperator(groupExpression.getOperator())
+ .filter(rule -> Objects.nonNull(rule) && rule.getPattern().matchOperator(groupExpression.getOperator())
Review Comment:
why add this? we have null in list?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ ExpressionUtils.extractConjunct(onPredicates.get()).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+ otherConjuncts.addAll(ExpressionUtils.extractConjunct(filterPredicates));
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_JOIN);
+ }
+
+ private Plan pushDownPredicate(LogicalBinaryPlan<LogicalJoin, GroupPlan, GroupPlan> joinPlan,
+ List<Expression> joinConditions, List<Expression> leftPredicates, List<Expression> rightPredicates) {
+
+ Expression left = ExpressionUtils.add(leftPredicates);
+ Expression right = ExpressionUtils.add(rightPredicates);
+ ExpressionRuleExecutor exprRewriter = new ExpressionRuleExecutor();
+ Plan leftPlan = joinPlan.left();
+ Plan rightPlan = joinPlan.right();
+ if (!left.equals(ExpressionUtils.TRUE_LITERAL)) {
+ leftPlan = plan(new LogicalFilter(exprRewriter.rewrite(left)), leftPlan);
+ }
+
+ if (!right.equals(ExpressionUtils.TRUE_LITERAL)) {
+ rightPlan = plan(new LogicalFilter(exprRewriter.rewrite(right)), rightPlan);
+ }
+
+ if (!joinConditions.isEmpty()) {
+ return plan(new LogicalJoin(joinPlan.getOperator().getJoinType(),
+ Optional.of(ExpressionUtils.add(joinConditions))), leftPlan, rightPlan);
+ }
+
+ return joinPlan.withChildren(Lists.newArrayList(leftPlan, rightPlan));
+ }
+
+ private Expression getJoinCondition(Expression predicate, List<Slot> leftOutput, List<Slot> rightOutput) {
+ if (!(predicate instanceof ComparisonPredicate)) {
+ return null;
+ }
+
+ ComparisonPredicate comparison = (ComparisonPredicate) predicate;
+
+ if (!(comparison.left() instanceof Slot) || !(comparison.right() instanceof Slot)) {
Review Comment:
maybe we need to support this situation:
```sql
on t1.k1 + 1 = t2.k2 - 1
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ ExpressionUtils.extractConjunct(onPredicates.get()).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+ otherConjuncts.addAll(ExpressionUtils.extractConjunct(filterPredicates));
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
Review Comment:
why slot is empty present this conjunct is both leftPredicate and rightPredicate?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ ExpressionUtils.extractConjunct(onPredicates.get()).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+ otherConjuncts.addAll(ExpressionUtils.extractConjunct(filterPredicates));
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
Review Comment:
expressions in output are not always slot and slot in predicate is a slot reference refer to expressions in child's output. So we should use ExprId in slot reference to judge which child it come from
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/CompoundPredicate.java:
##########
@@ -37,13 +39,31 @@ public CompoundPredicate(NodeType type, LEFT_CHILD_TYPE left, RIGHT_CHILD_TYPE r
super(type, left, right);
}
+ @Override
+ public String toString() {
+ return sql();
+ }
+
@Override
public String sql() {
String nodeType = getType().toString();
- return left().sql() + ' ' + nodeType + ' ' + right().sql();
+ return left() + " " + nodeType + " " + right();
Review Comment:
why remove '.sql()'?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.util;
+
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+
+import com.google.common.collect.Lists;
+
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Expression rewrite helper class.
+ */
+public class ExpressionUtils {
+
+ public static final Literal TRUE_LITERAL = new Literal(true);
Review Comment:
move to Literal maybe better
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
Review Comment:
currently, we only support equal on condition, add a TODO to support other on condition
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.util;
+
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+
+import com.google.common.collect.Lists;
+
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Expression rewrite helper class.
+ */
+public class ExpressionUtils {
+
+ public static final Literal TRUE_LITERAL = new Literal(true);
+ public static final Literal FALSE_LITERAL = new Literal(false);
+
+ public static boolean isConstant(Expression expr) {
+ return expr.isConstant();
+ }
+
+ public static List<Expression> extractConjunct(Expression expr) {
+ return extract(NodeType.AND, expr);
+ }
+
+
+ public static List<Expression> extractDisjunct(Expression expr) {
+ return extract(NodeType.OR, expr);
+ }
+
+ public static List<Expression> extract(CompoundPredicate expr) {
+ return extract(expr.getType(), expr);
+ }
+
+ private static List<Expression> extract(NodeType op, Expression expr) {
+ List<Expression> result = Lists.newArrayList();
+ extract(op, expr, result);
+ return result;
+ }
+
+ private static void extract(NodeType op, Expression expr, List<Expression> result) {
+ if (expr instanceof CompoundPredicate && expr.getType() == op) {
+ CompoundPredicate predicate = (CompoundPredicate) expr;
+ extract(op, predicate.left(), result);
+ extract(op, predicate.right(), result);
+ } else {
+ result.add(expr);
+ }
+ }
+
+
+ public static Expression add(List<Expression> expressions) {
+ return combine(NodeType.AND, expressions);
+ }
+
+ public static Expression add(Expression... expressions) {
+ return combine(NodeType.AND, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(Expression... expressions) {
+ return combine(NodeType.OR, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(List<Expression> expressions) {
+ return combine(NodeType.OR, expressions);
+ }
+
+ /**
+ * Use AND/OR to combine expressions together.
+ */
+ public static Expression combine(NodeType op, List<Expression> expressions) {
+
+ Objects.requireNonNull(expressions, "expressions is null");
+
+ if (expressions.size() == 0) {
+ if (op == NodeType.AND) {
+ return new Literal(true);
+ }
+ if (op == NodeType.OR) {
+ return new Literal(false);
+ }
+ }
+
+ if (expressions.size() == 1) {
+ return expressions.get(0);
+ }
+
+ List<Expression> distinctExpressions = Lists.newArrayList(new LinkedHashSet<>(expressions));
+ if (op == NodeType.AND) {
+ if (distinctExpressions.contains(FALSE_LITERAL)) {
+ return FALSE_LITERAL;
+ }
+ distinctExpressions = distinctExpressions.stream().filter(p -> !p.equals(TRUE_LITERAL))
+ .collect(Collectors.toList());
+ }
+
+ if (op == NodeType.OR) {
+ if (distinctExpressions.contains(TRUE_LITERAL)) {
+ return TRUE_LITERAL;
+ }
+ distinctExpressions = distinctExpressions.stream().filter(p -> !p.equals(FALSE_LITERAL))
+ .collect(Collectors.toList());
+ }
+
+ List<List<Expression>> partitions = Lists.partition(distinctExpressions, 2);
+ List<Expression> result = new LinkedList<>();
+
+ for (List<Expression> partition : partitions) {
+ if (partition.size() == 2) {
+ result.add(new CompoundPredicate(op, partition.get(0), partition.get(1)));
+ }
+ if (partition.size() == 1) {
+ result.add(partition.get(0));
+ }
+ }
+
+ return combine(op, result);
Review Comment:
how about just combine it one by one?
```java
Expression newExpression = distinctExpressions.stream()
.reduce((combined, expression) -> new CompoundPredicate<>(NodeType.AND, combined, expression)).get();
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ ExpressionUtils.extractConjunct(onPredicates.get()).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+ otherConjuncts.addAll(ExpressionUtils.extractConjunct(filterPredicates));
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_JOIN);
+ }
+
+ private Plan pushDownPredicate(LogicalBinaryPlan<LogicalJoin, GroupPlan, GroupPlan> joinPlan,
+ List<Expression> joinConditions, List<Expression> leftPredicates, List<Expression> rightPredicates) {
+
+ Expression left = ExpressionUtils.add(leftPredicates);
+ Expression right = ExpressionUtils.add(rightPredicates);
+ ExpressionRuleExecutor exprRewriter = new ExpressionRuleExecutor();
+ Plan leftPlan = joinPlan.left();
+ Plan rightPlan = joinPlan.right();
+ if (!left.equals(ExpressionUtils.TRUE_LITERAL)) {
+ leftPlan = plan(new LogicalFilter(exprRewriter.rewrite(left)), leftPlan);
+ }
+
+ if (!right.equals(ExpressionUtils.TRUE_LITERAL)) {
+ rightPlan = plan(new LogicalFilter(exprRewriter.rewrite(right)), rightPlan);
+ }
+
+ if (!joinConditions.isEmpty()) {
+ return plan(new LogicalJoin(joinPlan.getOperator().getJoinType(),
+ Optional.of(ExpressionUtils.add(joinConditions))), leftPlan, rightPlan);
+ }
+
+ return joinPlan.withChildren(Lists.newArrayList(leftPlan, rightPlan));
+ }
+
+ private Expression getJoinCondition(Expression predicate, List<Slot> leftOutput, List<Slot> rightOutput) {
+ if (!(predicate instanceof ComparisonPredicate)) {
+ return null;
+ }
+
+ ComparisonPredicate comparison = (ComparisonPredicate) predicate;
+
+ if (!(comparison.left() instanceof Slot) || !(comparison.right() instanceof Slot)) {
+ return null;
+ }
+
+ Slot left = (Slot) comparison.left();
+ Slot right = (Slot) comparison.right();
+
+ if (!leftOutput.contains(left)) {
+ Slot tmp = left;
+ left = right;
+ right = tmp;
+ }
+
+ if (leftOutput.contains(left) && rightOutput.contains(right)) {
+ return predicate;
+ }
Review Comment:
```suggestion
if ((leftOutput.contains(left) && rightOutput.contains(right))
|| (leftOutput.contains(right) && rightOutput.contains(left))) {
return predicate;
}
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.util;
+
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+
+import com.google.common.collect.Lists;
+
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Expression rewrite helper class.
+ */
+public class ExpressionUtils {
+
+ public static final Literal TRUE_LITERAL = new Literal(true);
+ public static final Literal FALSE_LITERAL = new Literal(false);
+
+ public static boolean isConstant(Expression expr) {
+ return expr.isConstant();
+ }
+
+ public static List<Expression> extractConjunct(Expression expr) {
+ return extract(NodeType.AND, expr);
+ }
+
+
+ public static List<Expression> extractDisjunct(Expression expr) {
+ return extract(NodeType.OR, expr);
+ }
+
+ public static List<Expression> extract(CompoundPredicate expr) {
+ return extract(expr.getType(), expr);
+ }
+
+ private static List<Expression> extract(NodeType op, Expression expr) {
+ List<Expression> result = Lists.newArrayList();
+ extract(op, expr, result);
+ return result;
+ }
+
+ private static void extract(NodeType op, Expression expr, List<Expression> result) {
+ if (expr instanceof CompoundPredicate && expr.getType() == op) {
+ CompoundPredicate predicate = (CompoundPredicate) expr;
+ extract(op, predicate.left(), result);
+ extract(op, predicate.right(), result);
+ } else {
+ result.add(expr);
+ }
+ }
+
+
+ public static Expression add(List<Expression> expressions) {
+ return combine(NodeType.AND, expressions);
+ }
+
+ public static Expression add(Expression... expressions) {
+ return combine(NodeType.AND, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(Expression... expressions) {
+ return combine(NodeType.OR, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(List<Expression> expressions) {
+ return combine(NodeType.OR, expressions);
+ }
+
+ /**
+ * Use AND/OR to combine expressions together.
+ */
+ public static Expression combine(NodeType op, List<Expression> expressions) {
+
+ Objects.requireNonNull(expressions, "expressions is null");
+
+ if (expressions.size() == 0) {
+ if (op == NodeType.AND) {
+ return new Literal(true);
+ }
+ if (op == NodeType.OR) {
+ return new Literal(false);
+ }
+ }
+
+ if (expressions.size() == 1) {
+ return expressions.get(0);
+ }
+
+ List<Expression> distinctExpressions = Lists.newArrayList(new LinkedHashSet<>(expressions));
Review Comment:
what is LinkedHashSet use for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r910652779
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_JOIN);
+ }
+
+ private Plan pushDownPredicate(LogicalBinaryPlan<LogicalJoin, GroupPlan, GroupPlan> joinPlan,
+ List<Expression> joinConditions, List<Expression> leftPredicates, List<Expression> rightPredicates) {
+
+ Expression left = ExpressionUtils.add(leftPredicates);
+ Expression right = ExpressionUtils.add(rightPredicates);
+ ExpressionRuleExecutor exprRewriter = new ExpressionRuleExecutor();
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] EmmyMiao87 merged pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 merged PR #10462:
URL: https://github.com/apache/doris/pull/10462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r911556425
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
Review Comment:
other cases will be added later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908182921
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.util;
+
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+
+import com.google.common.collect.Lists;
+
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Expression rewrite helper class.
+ */
+public class ExpressionUtils {
+
+ public static final Literal TRUE_LITERAL = new Literal(true);
+ public static final Literal FALSE_LITERAL = new Literal(false);
+
+ public static boolean isConstant(Expression expr) {
+ return expr.isConstant();
+ }
+
+ public static List<Expression> extractConjunct(Expression expr) {
+ return extract(NodeType.AND, expr);
+ }
+
+
+ public static List<Expression> extractDisjunct(Expression expr) {
+ return extract(NodeType.OR, expr);
+ }
+
+ public static List<Expression> extract(CompoundPredicate expr) {
+ return extract(expr.getType(), expr);
+ }
+
+ private static List<Expression> extract(NodeType op, Expression expr) {
+ List<Expression> result = Lists.newArrayList();
+ extract(op, expr, result);
+ return result;
+ }
+
+ private static void extract(NodeType op, Expression expr, List<Expression> result) {
+ if (expr instanceof CompoundPredicate && expr.getType() == op) {
+ CompoundPredicate predicate = (CompoundPredicate) expr;
+ extract(op, predicate.left(), result);
+ extract(op, predicate.right(), result);
+ } else {
+ result.add(expr);
+ }
+ }
+
+
+ public static Expression add(List<Expression> expressions) {
+ return combine(NodeType.AND, expressions);
+ }
+
+ public static Expression add(Expression... expressions) {
+ return combine(NodeType.AND, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(Expression... expressions) {
+ return combine(NodeType.OR, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(List<Expression> expressions) {
+ return combine(NodeType.OR, expressions);
+ }
+
+ /**
+ * Use AND/OR to combine expressions together.
+ */
+ public static Expression combine(NodeType op, List<Expression> expressions) {
+
+ Objects.requireNonNull(expressions, "expressions is null");
+
+ if (expressions.size() == 0) {
+ if (op == NodeType.AND) {
+ return new Literal(true);
+ }
+ if (op == NodeType.OR) {
+ return new Literal(false);
+ }
+ }
+
+ if (expressions.size() == 1) {
+ return expressions.get(0);
+ }
+
+ List<Expression> distinctExpressions = Lists.newArrayList(new LinkedHashSet<>(expressions));
Review Comment:
i got it, it is used for unique
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] EmmyMiao87 commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r910968327
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_JOIN);
+ }
+
+ private Plan pushDownPredicate(LogicalBinaryPlan<LogicalJoin, GroupPlan, GroupPlan> joinPlan,
+ List<Expression> joinConditions, List<Expression> leftPredicates, List<Expression> rightPredicates) {
+
+ Expression left = ExpressionUtils.add(leftPredicates);
+ Expression right = ExpressionUtils.add(rightPredicates);
+ ExpressionRuleExecutor exprRewriter = new ExpressionRuleExecutor();
Review Comment:
Please add a todo in here ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908206772
##########
fe/fe-core/src/test/java/org/apache/doris/nereids/util/ExpressionUtilsTest.java:
##########
@@ -0,0 +1,84 @@
+package org.apache.doris.nereids.util;
Review Comment:
miss license header
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/IterationVisitor.java:
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+/**
+ * Iterative traversal of an expression.
+ */
+public abstract class IterationVisitor<C> extends ExpressionVisitor<Void, C> {
+
+ @Override
+ public Void visit(Expression expr, C context) {
+ return expr.accept(this, context);
+ }
+
+ public Void visitNot(Not expr, C context) {
+ visit(expr.child(), context);
+ return null;
+ }
+
+ public Void visitGreaterThan(GreaterThan expr, C context) {
+ visit(expr.left(), context);
+ visit(expr.right(), context);
+ return null;
+ }
+
+ public Void visitEqualTo(EqualTo eq, C context) {
+ visit(eq.left(), context);
+ visit(eq.right(), context);
+ return null;
+ }
+
+ public Void visitCompoundPredicate(CompoundPredicate expr, C context) {
+ visit(expr.left(), context);
+ visit(expr.right(), context);
+ return null;
+ }
+
+ public Void visitLiteral(Literal literal, C context) {
+ return null;
+ }
+
+ public Void visitArithmetic(Arithmetic arithmetic, C context) {
Review Comment:
currently, Arithmetic could be unary and binary, so need to use Arithmetic#isBinary to check it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908117185
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.util;
+
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+
+import com.google.common.collect.Lists;
+
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Expression rewrite helper class.
+ */
+public class ExpressionUtils {
+
+ public static final Literal TRUE_LITERAL = new Literal(true);
+ public static final Literal FALSE_LITERAL = new Literal(false);
+
+ public static boolean isConstant(Expression expr) {
+ return expr.isConstant();
+ }
+
+ public static List<Expression> extractConjunct(Expression expr) {
+ return extract(NodeType.AND, expr);
+ }
+
+
+ public static List<Expression> extractDisjunct(Expression expr) {
+ return extract(NodeType.OR, expr);
+ }
+
+ public static List<Expression> extract(CompoundPredicate expr) {
+ return extract(expr.getType(), expr);
+ }
+
+ private static List<Expression> extract(NodeType op, Expression expr) {
+ List<Expression> result = Lists.newArrayList();
+ extract(op, expr, result);
+ return result;
+ }
+
+ private static void extract(NodeType op, Expression expr, List<Expression> result) {
+ if (expr instanceof CompoundPredicate && expr.getType() == op) {
+ CompoundPredicate predicate = (CompoundPredicate) expr;
+ extract(op, predicate.left(), result);
+ extract(op, predicate.right(), result);
+ } else {
+ result.add(expr);
+ }
+ }
+
+
+ public static Expression add(List<Expression> expressions) {
+ return combine(NodeType.AND, expressions);
+ }
+
+ public static Expression add(Expression... expressions) {
+ return combine(NodeType.AND, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(Expression... expressions) {
+ return combine(NodeType.OR, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(List<Expression> expressions) {
+ return combine(NodeType.OR, expressions);
+ }
+
+ /**
+ * Use AND/OR to combine expressions together.
+ */
+ public static Expression combine(NodeType op, List<Expression> expressions) {
+
+ Objects.requireNonNull(expressions, "expressions is null");
+
+ if (expressions.size() == 0) {
+ if (op == NodeType.AND) {
+ return new Literal(true);
+ }
+ if (op == NodeType.OR) {
+ return new Literal(false);
+ }
+ }
+
+ if (expressions.size() == 1) {
+ return expressions.get(0);
+ }
+
+ List<Expression> distinctExpressions = Lists.newArrayList(new LinkedHashSet<>(expressions));
Review Comment:
The combine repeated predicate makes no sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908036657
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ ExpressionUtils.extractConjunct(onPredicates.get()).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+ otherConjuncts.addAll(ExpressionUtils.extractConjunct(filterPredicates));
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
Review Comment:
expressions in output are not always slot and slot in predicate is a slot reference refer to expressions in child's output. So we should use ExprId in slot reference to judge which child it come from
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908110764
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/Job.java:
##########
@@ -57,7 +58,7 @@ public RuleSet getRuleSet() {
public List<Rule<NODE_TYPE>> getValidRules(GroupExpression groupExpression,
List<Rule<NODE_TYPE>> candidateRules) {
return candidateRules.stream()
- .filter(rule -> rule.getPattern().matchOperator(groupExpression.getOperator())
+ .filter(rule -> Objects.nonNull(rule) && rule.getPattern().matchOperator(groupExpression.getOperator())
Review Comment:
prevent null cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908172324
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.util;
+
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+
+import com.google.common.collect.Lists;
+
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Expression rewrite helper class.
+ */
+public class ExpressionUtils {
+
+ public static final Literal TRUE_LITERAL = new Literal(true);
+ public static final Literal FALSE_LITERAL = new Literal(false);
+
+ public static boolean isConstant(Expression expr) {
+ return expr.isConstant();
+ }
+
+ public static List<Expression> extractConjunct(Expression expr) {
+ return extract(NodeType.AND, expr);
+ }
+
+
+ public static List<Expression> extractDisjunct(Expression expr) {
+ return extract(NodeType.OR, expr);
+ }
+
+ public static List<Expression> extract(CompoundPredicate expr) {
+ return extract(expr.getType(), expr);
+ }
+
+ private static List<Expression> extract(NodeType op, Expression expr) {
+ List<Expression> result = Lists.newArrayList();
+ extract(op, expr, result);
+ return result;
+ }
+
+ private static void extract(NodeType op, Expression expr, List<Expression> result) {
+ if (expr instanceof CompoundPredicate && expr.getType() == op) {
+ CompoundPredicate predicate = (CompoundPredicate) expr;
+ extract(op, predicate.left(), result);
+ extract(op, predicate.right(), result);
+ } else {
+ result.add(expr);
+ }
+ }
+
+
+ public static Expression add(List<Expression> expressions) {
+ return combine(NodeType.AND, expressions);
+ }
+
+ public static Expression add(Expression... expressions) {
+ return combine(NodeType.AND, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(Expression... expressions) {
+ return combine(NodeType.OR, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(List<Expression> expressions) {
+ return combine(NodeType.OR, expressions);
+ }
+
+ /**
+ * Use AND/OR to combine expressions together.
+ */
+ public static Expression combine(NodeType op, List<Expression> expressions) {
+
+ Objects.requireNonNull(expressions, "expressions is null");
+
+ if (expressions.size() == 0) {
+ if (op == NodeType.AND) {
+ return new Literal(true);
+ }
+ if (op == NodeType.OR) {
+ return new Literal(false);
+ }
+ }
+
+ if (expressions.size() == 1) {
+ return expressions.get(0);
+ }
+
+ List<Expression> distinctExpressions = Lists.newArrayList(new LinkedHashSet<>(expressions));
+ if (op == NodeType.AND) {
+ if (distinctExpressions.contains(FALSE_LITERAL)) {
+ return FALSE_LITERAL;
+ }
+ distinctExpressions = distinctExpressions.stream().filter(p -> !p.equals(TRUE_LITERAL))
+ .collect(Collectors.toList());
+ }
+
+ if (op == NodeType.OR) {
+ if (distinctExpressions.contains(TRUE_LITERAL)) {
+ return TRUE_LITERAL;
+ }
+ distinctExpressions = distinctExpressions.stream().filter(p -> !p.equals(FALSE_LITERAL))
+ .collect(Collectors.toList());
+ }
+
+ List<List<Expression>> partitions = Lists.partition(distinctExpressions, 2);
+ List<Expression> result = new LinkedList<>();
+
+ for (List<Expression> partition : partitions) {
+ if (partition.size() == 2) {
+ result.add(new CompoundPredicate(op, partition.get(0), partition.get(1)));
+ }
+ if (partition.size() == 1) {
+ result.add(partition.get(0));
+ }
+ }
+
+ return combine(op, result);
Review Comment:
This is for now and will be revised later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908113857
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/CompoundPredicate.java:
##########
@@ -37,13 +39,31 @@ public CompoundPredicate(NodeType type, LEFT_CHILD_TYPE left, RIGHT_CHILD_TYPE r
super(type, left, right);
}
+ @Override
+ public String toString() {
+ return sql();
+ }
+
@Override
public String sql() {
String nodeType = getType().toString();
- return left().sql() + ' ' + nodeType + ' ' + right().sql();
+ return left() + " " + nodeType + " " + right();
Review Comment:
Some expressions implement toString but not implement toSql. Unified implementation of SQL () is better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908131718
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ ExpressionUtils.extractConjunct(onPredicates.get()).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+ otherConjuncts.addAll(ExpressionUtils.extractConjunct(filterPredicates));
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
Review Comment:
a predicate might be a constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908213281
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ ExpressionUtils.extractConjunct(onPredicates.get()).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+ otherConjuncts.addAll(ExpressionUtils.extractConjunct(filterPredicates));
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_JOIN);
+ }
+
+ private Plan pushDownPredicate(LogicalBinaryPlan<LogicalJoin, GroupPlan, GroupPlan> joinPlan,
+ List<Expression> joinConditions, List<Expression> leftPredicates, List<Expression> rightPredicates) {
+
+ Expression left = ExpressionUtils.add(leftPredicates);
+ Expression right = ExpressionUtils.add(rightPredicates);
+ ExpressionRuleExecutor exprRewriter = new ExpressionRuleExecutor();
+ Plan leftPlan = joinPlan.left();
+ Plan rightPlan = joinPlan.right();
+ if (!left.equals(ExpressionUtils.TRUE_LITERAL)) {
+ leftPlan = plan(new LogicalFilter(exprRewriter.rewrite(left)), leftPlan);
+ }
+
+ if (!right.equals(ExpressionUtils.TRUE_LITERAL)) {
+ rightPlan = plan(new LogicalFilter(exprRewriter.rewrite(right)), rightPlan);
+ }
+
+ if (!joinConditions.isEmpty()) {
+ return plan(new LogicalJoin(joinPlan.getOperator().getJoinType(),
+ Optional.of(ExpressionUtils.add(joinConditions))), leftPlan, rightPlan);
+ }
+
+ return joinPlan.withChildren(Lists.newArrayList(leftPlan, rightPlan));
+ }
+
+ private Expression getJoinCondition(Expression predicate, List<Slot> leftOutput, List<Slot> rightOutput) {
+ if (!(predicate instanceof ComparisonPredicate)) {
+ return null;
+ }
+
+ ComparisonPredicate comparison = (ComparisonPredicate) predicate;
+
+ if (!(comparison.left() instanceof Slot) || !(comparison.right() instanceof Slot)) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] EmmyMiao87 commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r910968848
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_JOIN);
+ }
+
+ private Plan pushDownPredicate(LogicalBinaryPlan<LogicalJoin, GroupPlan, GroupPlan> joinPlan,
+ List<Expression> joinConditions, List<Expression> leftPredicates, List<Expression> rightPredicates) {
+
+ Expression left = ExpressionUtils.add(leftPredicates);
+ Expression right = ExpressionUtils.add(rightPredicates);
+ ExpressionRuleExecutor exprRewriter = new ExpressionRuleExecutor();
+ Plan leftPlan = joinPlan.left();
Review Comment:
Please add todo in here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r911557867
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_JOIN);
+ }
+
+ private Plan pushDownPredicate(LogicalBinaryPlan<LogicalJoin, GroupPlan, GroupPlan> joinPlan,
+ List<Expression> joinConditions, List<Expression> leftPredicates, List<Expression> rightPredicates) {
+
+ Expression left = ExpressionUtils.add(leftPredicates);
+ Expression right = ExpressionUtils.add(rightPredicates);
+ ExpressionRuleExecutor exprRewriter = new ExpressionRuleExecutor();
+ Plan leftPlan = joinPlan.left();
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ Expression onCondition = Literal.TRUE_LITERAL;
+ if (onPredicates.isPresent()) {
+ onCondition = onPredicates.get();
+ }
+ ExpressionUtils.extractConjunct(ExpressionUtils.add(onCondition, filterPredicates)).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_JOIN);
+ }
+
+ private Plan pushDownPredicate(LogicalBinaryPlan<LogicalJoin, GroupPlan, GroupPlan> joinPlan,
+ List<Expression> joinConditions, List<Expression> leftPredicates, List<Expression> rightPredicates) {
+
+ Expression left = ExpressionUtils.add(leftPredicates);
+ Expression right = ExpressionUtils.add(rightPredicates);
+ ExpressionRuleExecutor exprRewriter = new ExpressionRuleExecutor();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r910653748
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908118263
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.util;
+
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+
+import com.google.common.collect.Lists;
+
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Expression rewrite helper class.
+ */
+public class ExpressionUtils {
+
+ public static final Literal TRUE_LITERAL = new Literal(true);
+ public static final Literal FALSE_LITERAL = new Literal(false);
+
+ public static boolean isConstant(Expression expr) {
+ return expr.isConstant();
+ }
+
+ public static List<Expression> extractConjunct(Expression expr) {
+ return extract(NodeType.AND, expr);
+ }
+
+
+ public static List<Expression> extractDisjunct(Expression expr) {
+ return extract(NodeType.OR, expr);
+ }
+
+ public static List<Expression> extract(CompoundPredicate expr) {
+ return extract(expr.getType(), expr);
+ }
+
+ private static List<Expression> extract(NodeType op, Expression expr) {
+ List<Expression> result = Lists.newArrayList();
+ extract(op, expr, result);
+ return result;
+ }
+
+ private static void extract(NodeType op, Expression expr, List<Expression> result) {
+ if (expr instanceof CompoundPredicate && expr.getType() == op) {
+ CompoundPredicate predicate = (CompoundPredicate) expr;
+ extract(op, predicate.left(), result);
+ extract(op, predicate.right(), result);
+ } else {
+ result.add(expr);
+ }
+ }
+
+
+ public static Expression add(List<Expression> expressions) {
+ return combine(NodeType.AND, expressions);
+ }
+
+ public static Expression add(Expression... expressions) {
+ return combine(NodeType.AND, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(Expression... expressions) {
+ return combine(NodeType.OR, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(List<Expression> expressions) {
+ return combine(NodeType.OR, expressions);
+ }
+
+ /**
+ * Use AND/OR to combine expressions together.
+ */
+ public static Expression combine(NodeType op, List<Expression> expressions) {
+
+ Objects.requireNonNull(expressions, "expressions is null");
+
+ if (expressions.size() == 0) {
+ if (op == NodeType.AND) {
+ return new Literal(true);
+ }
+ if (op == NodeType.OR) {
+ return new Literal(false);
+ }
+ }
+
+ if (expressions.size() == 1) {
+ return expressions.get(0);
+ }
+
+ List<Expression> distinctExpressions = Lists.newArrayList(new LinkedHashSet<>(expressions));
+ if (op == NodeType.AND) {
+ if (distinctExpressions.contains(FALSE_LITERAL)) {
+ return FALSE_LITERAL;
+ }
+ distinctExpressions = distinctExpressions.stream().filter(p -> !p.equals(TRUE_LITERAL))
+ .collect(Collectors.toList());
+ }
+
+ if (op == NodeType.OR) {
+ if (distinctExpressions.contains(TRUE_LITERAL)) {
+ return TRUE_LITERAL;
+ }
+ distinctExpressions = distinctExpressions.stream().filter(p -> !p.equals(FALSE_LITERAL))
+ .collect(Collectors.toList());
+ }
+
+ List<List<Expression>> partitions = Lists.partition(distinctExpressions, 2);
+ List<Expression> result = new LinkedList<>();
+
+ for (List<Expression> partition : partitions) {
+ if (partition.size() == 2) {
+ result.add(new CompoundPredicate(op, partition.get(0), partition.get(1)));
+ }
+ if (partition.size() == 1) {
+ result.add(partition.get(0));
+ }
+ }
+
+ return combine(op, result);
Review Comment:
nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908123194
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
+
+ List<Slot> leftInput = plan.child().left().getOutput();
+ List<Slot> rightInput = plan.child().right().getOutput();
+
+ List<Expression> joinConditions = Lists.newArrayList();
+ List<Expression> otherConjuncts = Lists.newArrayList();
+
+ ExpressionUtils.extractConjunct(onPredicates.get()).forEach(predicate -> {
+ if (Objects.nonNull(getJoinCondition(predicate, leftInput, rightInput))) {
+ joinConditions.add(predicate);
+ } else {
+ otherConjuncts.add(predicate);
+ }
+ });
+ otherConjuncts.addAll(ExpressionUtils.extractConjunct(filterPredicates));
+
+ List<Expression> leftPredicates = Lists.newArrayList();
+ List<Expression> rightPredicates = Lists.newArrayList();
+
+ for (Expression conjunct : otherConjuncts) {
+ Set<Slot> slots = SlotExtractor.extractSlot(conjunct);
+
+ if (slots.isEmpty()) {
+ leftPredicates.add(conjunct);
+ rightPredicates.add(conjunct);
+ continue;
+ }
+ if (leftInput.containsAll(slots)) {
+ leftPredicates.add(conjunct);
+ }
+ if (rightInput.containsAll(slots)) {
+ rightPredicates.add(conjunct);
+ }
+ }
+ otherConjuncts.removeAll(leftPredicates);
+ otherConjuncts.removeAll(rightPredicates);
+
+ joinConditions.addAll(otherConjuncts);
+
+ return pushDownPredicate(plan.child(), joinConditions, leftPredicates, rightPredicates);
+ }).toRule(RuleType.PUSH_DOWN_PREDICATE_THROUGH_JOIN);
+ }
+
+ private Plan pushDownPredicate(LogicalBinaryPlan<LogicalJoin, GroupPlan, GroupPlan> joinPlan,
+ List<Expression> joinConditions, List<Expression> leftPredicates, List<Expression> rightPredicates) {
+
+ Expression left = ExpressionUtils.add(leftPredicates);
+ Expression right = ExpressionUtils.add(rightPredicates);
+ ExpressionRuleExecutor exprRewriter = new ExpressionRuleExecutor();
+ Plan leftPlan = joinPlan.left();
+ Plan rightPlan = joinPlan.right();
+ if (!left.equals(ExpressionUtils.TRUE_LITERAL)) {
+ leftPlan = plan(new LogicalFilter(exprRewriter.rewrite(left)), leftPlan);
+ }
+
+ if (!right.equals(ExpressionUtils.TRUE_LITERAL)) {
+ rightPlan = plan(new LogicalFilter(exprRewriter.rewrite(right)), rightPlan);
+ }
+
+ if (!joinConditions.isEmpty()) {
+ return plan(new LogicalJoin(joinPlan.getOperator().getJoinType(),
+ Optional.of(ExpressionUtils.add(joinConditions))), leftPlan, rightPlan);
+ }
+
+ return joinPlan.withChildren(Lists.newArrayList(leftPlan, rightPlan));
+ }
+
+ private Expression getJoinCondition(Expression predicate, List<Slot> leftOutput, List<Slot> rightOutput) {
+ if (!(predicate instanceof ComparisonPredicate)) {
+ return null;
+ }
+
+ ComparisonPredicate comparison = (ComparisonPredicate) predicate;
+
+ if (!(comparison.left() instanceof Slot) || !(comparison.right() instanceof Slot)) {
+ return null;
+ }
+
+ Slot left = (Slot) comparison.left();
+ Slot right = (Slot) comparison.right();
+
+ if (!leftOutput.contains(left)) {
+ Slot tmp = left;
+ left = right;
+ right = tmp;
+ }
+
+ if (leftOutput.contains(left) && rightOutput.contains(right)) {
+ return predicate;
+ }
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908191708
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ */
+public class PushPredicateThroughJoin extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule<Plan> build() {
+ return logicalFilter(logicalJoin()).then(plan -> {
+
+ Expression filterPredicates = plan.operator.getPredicates();
+ Optional<Expression> onPredicates = plan.child().operator.getCondition();
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908129889
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/GroupExpressionMatching.java:
##########
@@ -133,6 +133,12 @@ private void assembleAllCombinationPlanTree(Plan root, Pattern<Plan, Plan> rootP
int[] childrenPlanIndex = new int[childrenPlans.size()];
int offset = 0;
+ for (List<Plan> plan : childrenPlans) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908181544
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/CompoundPredicate.java:
##########
@@ -37,13 +39,31 @@ public CompoundPredicate(NodeType type, LEFT_CHILD_TYPE left, RIGHT_CHILD_TYPE r
super(type, left, right);
}
+ @Override
+ public String toString() {
+ return sql();
+ }
+
@Override
public String sql() {
String nodeType = getType().toString();
- return left().sql() + ' ' + nodeType + ' ' + right().sql();
+ return left() + " " + nodeType + " " + right();
Review Comment:
ok, i will record as a TODO in doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r908120493
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.util;
+
+import org.apache.doris.nereids.trees.NodeType;
+import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+
+import com.google.common.collect.Lists;
+
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Expression rewrite helper class.
+ */
+public class ExpressionUtils {
+
+ public static final Literal TRUE_LITERAL = new Literal(true);
+ public static final Literal FALSE_LITERAL = new Literal(false);
+
+ public static boolean isConstant(Expression expr) {
+ return expr.isConstant();
+ }
+
+ public static List<Expression> extractConjunct(Expression expr) {
+ return extract(NodeType.AND, expr);
+ }
+
+
+ public static List<Expression> extractDisjunct(Expression expr) {
+ return extract(NodeType.OR, expr);
+ }
+
+ public static List<Expression> extract(CompoundPredicate expr) {
+ return extract(expr.getType(), expr);
+ }
+
+ private static List<Expression> extract(NodeType op, Expression expr) {
+ List<Expression> result = Lists.newArrayList();
+ extract(op, expr, result);
+ return result;
+ }
+
+ private static void extract(NodeType op, Expression expr, List<Expression> result) {
+ if (expr instanceof CompoundPredicate && expr.getType() == op) {
+ CompoundPredicate predicate = (CompoundPredicate) expr;
+ extract(op, predicate.left(), result);
+ extract(op, predicate.right(), result);
+ } else {
+ result.add(expr);
+ }
+ }
+
+
+ public static Expression add(List<Expression> expressions) {
+ return combine(NodeType.AND, expressions);
+ }
+
+ public static Expression add(Expression... expressions) {
+ return combine(NodeType.AND, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(Expression... expressions) {
+ return combine(NodeType.OR, Lists.newArrayList(expressions));
+ }
+
+ public static Expression or(List<Expression> expressions) {
+ return combine(NodeType.OR, expressions);
+ }
+
+ /**
+ * Use AND/OR to combine expressions together.
+ */
+ public static Expression combine(NodeType op, List<Expression> expressions) {
+
+ Objects.requireNonNull(expressions, "expressions is null");
+
+ if (expressions.size() == 0) {
+ if (op == NodeType.AND) {
+ return new Literal(true);
+ }
+ if (op == NodeType.OR) {
+ return new Literal(false);
+ }
+ }
+
+ if (expressions.size() == 1) {
+ return expressions.get(0);
+ }
+
+ List<Expression> distinctExpressions = Lists.newArrayList(new LinkedHashSet<>(expressions));
+ if (op == NodeType.AND) {
+ if (distinctExpressions.contains(FALSE_LITERAL)) {
+ return FALSE_LITERAL;
+ }
+ distinctExpressions = distinctExpressions.stream().filter(p -> !p.equals(TRUE_LITERAL))
+ .collect(Collectors.toList());
+ }
+
+ if (op == NodeType.OR) {
+ if (distinctExpressions.contains(TRUE_LITERAL)) {
+ return TRUE_LITERAL;
+ }
+ distinctExpressions = distinctExpressions.stream().filter(p -> !p.equals(FALSE_LITERAL))
+ .collect(Collectors.toList());
+ }
+
+ List<List<Expression>> partitions = Lists.partition(distinctExpressions, 2);
+ List<Expression> result = new LinkedList<>();
+
+ for (List<Expression> partition : partitions) {
+ if (partition.size() == 2) {
+ result.add(new CompoundPredicate(op, partition.get(0), partition.get(1)));
+ }
+ if (partition.size() == 1) {
+ result.add(partition.get(0));
+ }
+ }
+
+ return combine(op, result);
Review Comment:
It should also work. It's easier to implement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] qzsee commented on a diff in pull request #10462: [Enhancement](Nereids) push down predicate through join
Posted by GitBox <gi...@apache.org>.
qzsee commented on code in PR #10462:
URL: https://github.com/apache/doris/pull/10462#discussion_r910653567
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
Review Comment:
Predicate push-down needs to be discussed separately
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushPredicateThroughJoin.java:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.operators.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.operators.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.expression.rewrite.ExpressionRuleExecutor;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Literal;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotExtractor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalBinaryPlan;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push the predicate in the LogicalFilter or LogicalJoin to the join children.
+ * todo: Now, only support eq on condition, support other on condition later
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org