You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by "silundong (via GitHub)" <gi...@apache.org> on 2025/05/20 09:09:45 UTC
[PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
silundong opened a new pull request, #4392:
URL: https://github.com/apache/calcite/pull/4392
Add conflict detection algorithm CD-C to DPhyp so that it can handle various join types. See https://issues.apache.org/jira/browse/CALCITE-7029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2111464919
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperGraph.java:
##########
@@ -401,69 +453,67 @@ public RexNode extractJoinCond(RelNode left, RelNode right, List<HyperEdge> edge
};
for (HyperEdge edge : edges) {
- RexNode inputRefCond = edge.getCondition().accept(inputName2InputRefShuttle);
+ RexNode inputRefCond = edge.getCondition().accept(shuttle);
joinConds.add(inputRefCond);
}
- return RexUtil.composeConjunction(left.getCluster().getRexBuilder(), joinConds);
+ return RexUtil.composeConjunction(getCluster().getRexBuilder(), joinConds);
}
/**
- * Before starting enumeration, add Project on every input, make all field name unique.
- * Convert the HyperEdge condition from RexInputRef to RexInputFieldName
+ * Restore the projection order of the final result to the original plan.
+ *
+ * @param resultOrder the node order of the final result
+ * @param rowTypeList rowType of the final result
+ * @return list of RexInputRef
*/
- public void convertHyperEdgeCond(RelBuilder builder) {
- int fieldIndex = 0;
- List<RelDataTypeField> fieldList = rowType.getFieldList();
- for (int nodeIndex = 0; nodeIndex < inputs.size(); nodeIndex++) {
- RelNode input = inputs.get(nodeIndex);
- List<RexNode> projects = new ArrayList<>();
- List<String> names = new ArrayList<>();
- for (int i = 0; i < input.getRowType().getFieldCount(); i++) {
- projects.add(
- new RexInputRef(
- i,
- fieldList.get(fieldIndex).getType()));
- names.add(fieldList.get(fieldIndex).getName());
- fieldIndex++;
+ public List<RexNode> restoreProjectionOrder(
+ ImmutableList<Integer> resultOrder,
+ List<RelDataTypeField> rowTypeList) {
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
+ int fieldCount = 0;
+ for (int resultIndex : resultOrder) {
+ relativePositionInNode.put(resultIndex, fieldCount);
+ if (!LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(resultIndex))) {
+ fieldCount += inputs.get(resultIndex).getRowType().getFieldCount();
}
-
- builder.push(input)
- .project(projects, names, true);
- replaceInput(nodeIndex, builder.build());
}
-
- RexShuttle inputRef2inputNameShuttle = new RexShuttle() {
- @Override public RexNode visitInputRef(RexInputRef inputRef) {
- int index = inputRef.getIndex();
- return new RexInputFieldName(
- fieldList.get(index).getName(),
- fieldList.get(index).getType());
+ List<RexNode> projects = new ArrayList<>();
+ for (int inputIndex = 0; inputIndex < inputs.size(); inputIndex++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(inputIndex))) {
+ continue;
}
- };
- for (int i = 0; i < edges.size(); i++) {
- HyperEdge edge = edges.get(i);
- RexNode convertCond = edge.getCondition().accept(inputRef2inputNameShuttle);
- HyperEdge convertEdge =
- new HyperEdge(
- edge.getLeftNodeBitmap(),
- edge.getRightNodeBitmap(),
- edge.getJoinType(),
- convertCond);
- edges.set(i, convertEdge);
+ for (int i = 0; i < inputs.get(inputIndex).getRowType().getFieldCount(); i++) {
+ Integer fieldOffset = relativePositionInNode.get(inputIndex);
+ if (fieldOffset == null) {
+ throw new DpHyp.DphypOrHyperGraphException(
+ "The result order loses the " + inputIndex + "-th input");
+ }
+ int inputRef = i + fieldOffset;
+ projects.add(
+ new RexInputRef(inputRef, rowTypeList.get(inputRef).getType()));
+ }
}
+ return projects;
}
/**
* Adjusting RexInputRef in enumeration process is too complicated,
- * so use unique name replace input ref.
- * Before starting enumeration, convert RexInputRef to RexInputFieldName.
- * When connect csgcmp to Join, convert RexInputFieldName to RexInputRef.
+ * so use node index and relative position of field in node replace RexInputRef.
+ * When build hyper graph, convert RexInputRef to RexNodeAndFieldIndex.
+ * When connect csgcmp to Join, convert RexNodeAndFieldIndex to RexInputRef.
*/
- private static class RexInputFieldName extends RexVariable {
+ static class RexNodeAndFieldIndex extends RexVariable {
Review Comment:
In the process of constructing a hypergraph, it needs to be generated before `new HyperGraph()`, so I define it as static.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2111462658
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -37,7 +40,12 @@ public class DpHyp {
private final HyperGraph hyperGraph;
- private final HashMap<Long, RelNode> dpTable;
+ private final Map<Long, RelNode> dpTable;
+
+ // record the node order corresponding to the best subgraph, which is used to convert
+ // the RexNodeAndFieldIndex in hyperedge to the RexInputRef in join condition, and permute
+ // final result
+ private final Map<Long, ImmutableList<Integer>> resultInputOrder;
Review Comment:
resultInputOrder is the same size as dpTable. I think its size is equal to the number of connected components in the hypergraph. This can indeed become very large. Is it OK to check its size before putting data into it during enumeration (it cannot exceed a certain value, otherwise the enumeration ends)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2099657997
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * On the correct and complete enumeration of the core search space.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
Review Comment:
maybe can create a new jira for this TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2115110360
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
Review Comment:
His link is readable by anyone but may disappear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-3001423094
From my pov this is ready after a few minor changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-3003359856
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392) **Quality Gate passed**
Issues
 [13 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [74.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-3006639079
All reviewers approved this PR, I will merge it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2097517846
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -1526,6 +1526,33 @@ LogicalJoin(condition=[=($1, $12)], joinType=[semi])
LogicalTableScan(table=[[scott, DEPT]])
LogicalTableScan(table=[[scott, EMP]])
LogicalTableScan(table=[[scott, BONUS]])
+]]>
+ </Resource>
+ </TestCase>
+ <TestCase name="testCartesianProductForDphyp">
+ <Resource name="sql">
+ <![CDATA[select emp.empno from emp_address, emp, emp_b, dept where emp_address.empno = emp.empno and dept.deptno = emp.deptno]]>
+ </Resource>
+ <Resource name="planBefore">
+ <![CDATA[
+LogicalProject(EMPNO=[$3])
+ HyperGraph(edges=[{0}——[INNER, =(vertex(0)_field(0), vertex(1)_field(0))]——{1},{0, 1}——[INNER, true]——{2},{1}——[INNER, =(vertex(3)_field(0), vertex(1)_field(7))]——{3}])
Review Comment:
To be honest, I don't understand this algorithm, but I think the new plan can be added to a java doc to facilitate other people to review it.
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -17826,16 +17911,16 @@ LogicalProject(EMPNO=[$0])
</Resource>
<Resource name="planAfter">
<![CDATA[
-LogicalProject(EMPNO=[$19])
- LogicalJoin(condition=[AND(=($19, $0), =(+($8, $0), +($17, $13)))], joinType=[inner])
- LogicalTableScan(table=[[CATALOG, SALES, EMP_ADDRESS]])
+LogicalProject(EMPNO=[$16])
Review Comment:
The new PR changes the order of the left and right tables. Is this as expected?
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
Review Comment:
Can you please provide a link to the paper?
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * On the correct and complete enumeration of the core search space.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
+ List<HyperEdge> leftSubEdges,
+ List<HyperEdge> rightSubEdges,
+ JoinRelType joinType) {
+ Map<Long, Long> conflictRules = new HashMap<>();
+ for (HyperEdge leftSubEdge : leftSubEdges) {
+ if (!isAssociative(leftSubEdge.getJoinType(), joinType)) {
+ if (leftSubEdge.getLeftNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ leftSubEdge.getInitialRightNodeBits(),
+ leftSubEdge.getLeftNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ leftSubEdge.getInitialRightNodeBits(),
+ leftSubEdge.getInitialLeftNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+
+ }
+ if (!isLeftAsscom(leftSubEdge.getJoinType(), joinType)) {
+ if (leftSubEdge.getRightNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ leftSubEdge.getInitialLeftNodeBits(),
+ leftSubEdge.getRightNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ leftSubEdge.getInitialLeftNodeBits(),
+ leftSubEdge.getInitialRightNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ }
+
+ for (HyperEdge rightSubEdge : rightSubEdges) {
+ if (!isAssociative(joinType, rightSubEdge.getJoinType())) {
+ if (rightSubEdge.getRightNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ rightSubEdge.getInitialLeftNodeBits(),
+ rightSubEdge.getRightNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ rightSubEdge.getInitialLeftNodeBits(),
+ rightSubEdge.getInitialRightNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ if (!isRightAsscom(joinType, rightSubEdge.getJoinType())) {
+ if (rightSubEdge.getLeftNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ rightSubEdge.getInitialRightNodeBits(),
+ rightSubEdge.getLeftNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ rightSubEdge.getInitialRightNodeBits(),
+ rightSubEdge.getInitialLeftNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ }
+ return conflictRules;
+ }
+
+ /**
+ * For conflict rule T1 -> T2, if T1 and tes have intersection, we can add T2 into tes and remove
+ * this rule. See section 5.5 in paper.
+ *
+ * @param tes tes
+ * @param conflictRulesAfterAbsorb conflict rules after absorbing
+ * @param initialconflictRules conflict rules before absorbing
+ * @return tes after absorbing conflict rules
+ */
+ public static long absorbConflictRulesIntoTES(
+ long tes,
+ Map<Long, Long> conflictRulesAfterAbsorb,
+ Map<Long, Long> initialconflictRules) {
+ for (Map.Entry<Long, Long> rule : initialconflictRules.entrySet()) {
+ if (LongBitmap.isOverlap(tes, rule.getKey())) {
+ tes |= rule.getValue();
+ continue;
+ }
+ conflictRulesAfterAbsorb.put(rule.getKey(), rule.getValue());
+ }
+ return tes;
+ }
+
+ /**
+ * Check whether the operator is applicable through the conflict rule.
+ *
+ * @param subGraph table set of the current join operator
+ * @param edges hyper edges with conflict rules
+ * @return true if the operator is applicable
+ */
+ public static boolean applicable(long subGraph, List<HyperEdge> edges) {
+ for (HyperEdge edge : edges) {
+ for (Map.Entry<Long, Long> rule : edge.getConflictRules().entrySet()) {
+ if (LongBitmap.isOverlap(subGraph, rule.getKey())
+ && !LongBitmap.isSubSet(rule.getValue(), subGraph)) {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ public static boolean isCommutative(JoinRelType operator) {
+ return operator == JoinRelType.INNER || operator == JoinRelType.FULL;
+ }
+
+ public static boolean isAssociative(JoinRelType operatorA, JoinRelType operatorB) {
+ Integer indexA = INDEX_OF_TABLE.get(operatorA);
Review Comment:
The first few lines of the three methods are the same. Can the three methods be combined?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2114505648
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperGraph.java:
##########
@@ -401,69 +453,67 @@ public RexNode extractJoinCond(RelNode left, RelNode right, List<HyperEdge> edge
};
for (HyperEdge edge : edges) {
- RexNode inputRefCond = edge.getCondition().accept(inputName2InputRefShuttle);
+ RexNode inputRefCond = edge.getCondition().accept(shuttle);
joinConds.add(inputRefCond);
}
- return RexUtil.composeConjunction(left.getCluster().getRexBuilder(), joinConds);
+ return RexUtil.composeConjunction(getCluster().getRexBuilder(), joinConds);
Review Comment:
then maybe you can add some tests with such predicates; I am curious how the hypergraphs will look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2137472098
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,104 +75,160 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new AssertionError("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new AssertionError("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new AssertionError("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = initialLeftNodeBits;
+ rightEndpoint = initialRightNodeBits;
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
Review Comment:
a
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,104 +75,160 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new AssertionError("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new AssertionError("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new AssertionError("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = initialLeftNodeBits;
+ rightEndpoint = initialRightNodeBits;
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
Review Comment:
a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2111467661
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperGraph.java:
##########
@@ -401,69 +453,67 @@ public RexNode extractJoinCond(RelNode left, RelNode right, List<HyperEdge> edge
};
for (HyperEdge edge : edges) {
- RexNode inputRefCond = edge.getCondition().accept(inputName2InputRefShuttle);
+ RexNode inputRefCond = edge.getCondition().accept(shuttle);
joinConds.add(inputRefCond);
}
- return RexUtil.composeConjunction(left.getCluster().getRexBuilder(), joinConds);
+ return RexUtil.composeConjunction(getCluster().getRexBuilder(), joinConds);
Review Comment:
No, there is no constraint on the original join condition in the code. When constructing the hypergraph in the `JoinToHyperGraphRule`, for the original Join, I separated its join condition according to `AND`, each item can be used as a hyperedge, and I replaced `RexInputRef` in the expression with `RexNodeAndFieldIndex`. When enumerating the csg-cmp pair, I need to find all the hyperedges connecting them, restore the `RexNodeAndFieldIndex` in it to `RexInputRef`, and then form a conjunction as the join condition to construct the Join.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2151880132
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -77,9 +83,13 @@ public DpHyp(HyperGraph hyperGraph, RelBuilder builder, RelMetadataQuery relMeta
* in the {@link DpHyp#dpTable}.
*/
public void startEnumerateJoin() {
+ LOGGER.debug("Start dphyp enumeration.");
Review Comment:
I think this log is not necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2979815454
Please resolve the conflicts of RelOptRulesTest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2115746015
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,104 +75,160 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
Review Comment:
plz add comments for this RexShuttle
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,104 +75,160 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new AssertionError("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new AssertionError("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new AssertionError("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = initialLeftNodeBits;
+ rightEndpoint = initialRightNodeBits;
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
Review Comment:
https://github.com/cockroachdb/cockroach/blob/475172846823a877e07eabb5e8da1a996e510d17/pkg/sql/opt/xform/join_order_builder.go#L438
For any given non-inner join, exactly one edge is constructed.
do we have same consider?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2999039884
Very good test, I think it can reach a state where merging is possible. There are already over 100 comments. Please wait for the opinions of @mihaibudiu and @suibianwanwank.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2915778540
> > I haven't really checked that the algorithm is correctly implemented, I left only comments about the code structure. I think this needs way more tests, even if they are slow tests. You need for example to test on plans with lots of joins and check that all the orders generated are valid and generate the same result, even if they are not optimal. This will exercise the code much more deeply than the tests you have now.
>
> Thank you very much for your detailed review. I have replied to some comments and hope to discuss further with you. I will update the relevant code for the other comments that I have not replied to.
>
> In addition, when you say more tests, you mean that I should add some test cases in `RelOptRulesTest`, which have many tables involved in join graph, although the enumeration process may be slow in this case. Am I understanding it correctly?
If they are slow, these extra tests can be marked as "slow tests" and they won't be run all the time in CI (only when the "slow test" label is present).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2137072447
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -173,42 +193,97 @@ private void enumerateCmpRec(long csg, long cmp, long forbidden) {
private void emitCsgCmp(long csg, long cmp, List<HyperEdge> edges) {
RelNode child1 = dpTable.get(csg);
RelNode child2 = dpTable.get(cmp);
- if (child1 == null || child2 == null) {
- throw new IllegalArgumentException(
- "csg and cmp were not enumerated in the previous dp process");
- }
+ ImmutableList<HyperGraph.NodeState> csgOrder = resultInputOrder.get(csg);
+ ImmutableList<HyperGraph.NodeState> cmpOrder = resultInputOrder.get(cmp);
+ assert child1 != null && child2 != null && csgOrder != null && cmpOrder != null;
+ assert Long.bitCount(csg) == csgOrder.size() && Long.bitCount(cmp) == cmpOrder.size();
JoinRelType joinType = hyperGraph.extractJoinType(edges);
if (joinType == null) {
return;
}
- RexNode joinCond1 = hyperGraph.extractJoinCond(child1, child2, edges);
+ // verify whether the subgraph is legal by using the conflict rules in hyperedges
+ if (!hyperGraph.applicable(csg | cmp, edges)) {
+ return;
+ }
+
+ List<HyperGraph.NodeState> unionOrder = new ArrayList<>(csgOrder);
+ unionOrder.addAll(cmpOrder);
+ // build join condition from hyperedges. e.g.
Review Comment:
Could you explain what you are going to do with a sql or plan? These are a bit difficult to understand for people who don't know much about the paper, and the same for other places where formulas are used as comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2917082218
> more tests
No, I don't mean RelOptRulesTests. You need something that looks like unit tests for your code, not for Calcite. For example, when you enumerate candidate plans, it would be nice to validate that these plans are all valid and generate the same result, even if they are not optimal. (Checking that the result is correct may not be easy, though, given the current testing infrastructure.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2098009043
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -17826,16 +17911,16 @@ LogicalProject(EMPNO=[$0])
</Resource>
<Resource name="planAfter">
<![CDATA[
-LogicalProject(EMPNO=[$19])
- LogicalJoin(condition=[AND(=($19, $0), =(+($8, $0), +($17, $13)))], joinType=[inner])
- LogicalTableScan(table=[[CATALOG, SALES, EMP_ADDRESS]])
+LogicalProject(EMPNO=[$16])
Review Comment:
I think this is caused by double precision.
Before explaining this problem, I need to explain that in the previous pr, in order to conveniently represent `RexInputRef` during enumeration, a unique alias will be generated for each input before enumeration (that is, an additional Project operator will be generated).
However, relying on strings and input rowType will cause trouble when processing semi/anti joins (the right child will not be projected). Therefore, in this PR, I changed it to use node index and the relative position of the field in the node to represent `RexInputRef`.
So, even if the join order is essentially the same, the cost value of the plan in this pr and the previous pr is different during enumeration (the previous pr has an additional Project).
Specifically,
In the previous pr:
t1 join t2 cost=154.21564(with additional Project)
t2 join t1 cost=154.21564(with additional Project) **_winner_** (see `chooseBetterPlan` in DpHyp.java)
In this pr:
t1 join t2 cost=92.21564 **_winner_**
t2 join t1 cost=92.21564000000001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2098033495
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -17826,16 +17911,16 @@ LogicalProject(EMPNO=[$0])
</Resource>
<Resource name="planAfter">
<![CDATA[
-LogicalProject(EMPNO=[$19])
- LogicalJoin(condition=[AND(=($19, $0), =(+($8, $0), +($17, $13)))], joinType=[inner])
- LogicalTableScan(table=[[CATALOG, SALES, EMP_ADDRESS]])
+LogicalProject(EMPNO=[$16])
Review Comment:
Thanks for your detailed explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2116144373
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,104 +75,160 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new AssertionError("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new AssertionError("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new AssertionError("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = initialLeftNodeBits;
+ rightEndpoint = initialRightNodeBits;
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
Review Comment:
Thank you very much for pointing this out! This is indeed a bug in the current code. Inner join can be rewritten as cross product + where. Due to this property, inner join condition can be split by conjunction and freely combine with other inner join conditions. This privilege cannot be extended to non-inner joins, so we cannot split the condition of non-inner joins, that is, only generate one edge.
The following example may be more clear:
```
t1
id name
1 t1
2 t1
t2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2911998814
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [62.6% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2115111803
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2103854944
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,106 +75,205 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ rightEndpoint = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
- edges,
+ notProjectInputs,
+ leftSubEdges,
origJoin.getRowType());
call.transformTo(result);
}
- private static HyperEdge adjustNodeBit(HyperEdge hyperEdge, int nodeOffset, int fieldOffset) {
- RexNode newCondition = RexUtil.shift(hyperEdge.getCondition(), fieldOffset);
+ private static HyperEdge adjustNodeBit(HyperEdge hyperEdge, int nodeOffset) {
+ RexShuttle shiftNodeIndexShuttle = new RexShuttle() {
+ @Override protected List<RexNode> visitList(
+ List<? extends RexNode> exprs,
+ boolean @Nullable [] update) {
+ ImmutableList.Builder<RexNode> clonedOperands = ImmutableList.builder();
+ for (RexNode operand : exprs) {
+ RexNode clonedOperand;
+ if (operand instanceof HyperGraph.RexNodeAndFieldIndex) {
+ clonedOperand =
+ new HyperGraph.RexNodeAndFieldIndex(
+ ((HyperGraph.RexNodeAndFieldIndex) operand).nodeIndex + nodeOffset,
+ ((HyperGraph.RexNodeAndFieldIndex) operand).fieldIndex,
+ ((HyperGraph.RexNodeAndFieldIndex) operand).getName(),
+ operand.getType());
+ } else {
+ clonedOperand = operand.accept(this);
+ }
+ if ((clonedOperand != operand) && (update != null)) {
+ update[0] = true;
+ }
+ clonedOperands.add(clonedOperand);
+ }
+ return clonedOperands.build();
+ }
+ };
+ RexNode newCondition = hyperEdge.getCondition().accept(shiftNodeIndexShuttle);
+ Map<Long, Long> newConflictRules = new HashMap<>();
+ for (Map.Entry<Long, Long> entry : hyperEdge.getConflictRules().entrySet()) {
+ newConflictRules.put(entry.getKey() << nodeOffset, entry.getValue() << nodeOffset);
+ }
return new HyperEdge(
- hyperEdge.getLeftNodeBitmap() << nodeOffset,
- hyperEdge.getRightNodeBitmap() << nodeOffset,
+ hyperEdge.getLeftEndpoint() << nodeOffset,
+ hyperEdge.getRightEndpoint() << nodeOffset,
+ hyperEdge.getLeftNodeUsedInPredicate() << nodeOffset,
+ hyperEdge.getRightNodeUsedInPredicate() << nodeOffset,
+ newConflictRules,
+ hyperEdge.getInitialLeftNodeBits() << nodeOffset,
+ hyperEdge.getInitialRightNodeBits() << nodeOffset,
hyperEdge.getJoinType(),
newCondition);
}
+ private static boolean unSupportedJoinType(JoinRelType joinType) {
+ return joinType == JoinRelType.RIGHT
Review Comment:
This is because the paper did not consider this type. There is no right join in the associative, left-asscom and right-asscom tables, so I cannot calculate the conflict rules for right join. In addition, as you said, before building the hypergraph, user can convert the right join to left join in advance. Therefore, I exclude the right join here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2893622764
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [62.6% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2164594004
##########
core/src/test/java/org/apache/calcite/test/DphypJoinReorderTest.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.test;
+
+import org.apache.calcite.DataContexts;
+import org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.calcite.adapter.enumerable.EnumerableInterpretable;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRules;
+import org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRules;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.hep.HepMatchOrder;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.plan.hep.HepProgramBuilder;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.rules.DpHyp;
+import org.apache.calcite.rel.rules.HyperGraph;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.runtime.Bindable;
+import org.apache.calcite.tools.RelBuilder;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+public class DphypJoinReorderTest {
+
+ @Test void testInnerLeftSemiJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // left join t3 on id2=id3
+ // where exists (select * from t4 where id3=id4)
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.LEFT,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 1, "Tom", 3, "Li")
+ .semiJoin(
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult = "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testChainInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id2=id3
+ // inner join t4 on id3=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, null, 3, "Li")
+ .values(new String[]{"id2", "name2"}, 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=2; name1=null; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testStarInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id1=id3
+ // inner join t4 on id1=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy", 1, "Tom")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom; id4=1; name4=Tom\n"
+ + "id1=2; name1=Lucy; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ /**
+ * Run the customized plan and check whether its execution results meet expectations (ignore row order).
+ *
+ * @param customPlanFunction a function that accepts one RelBuilder and produces a RelNode
+ * @param expectedResult expected result
+ */
+ void run(Function<RelBuilder, RelNode> customPlanFunction, String expectedResult) {
+ VolcanoPlanner volcanoPlanner = new VolcanoPlanner();
+ volcanoPlanner.addRelTraitDef(ConventionTraitDef.INSTANCE);
+ volcanoPlanner.addRelTraitDef(RelCollationTraitDef.INSTANCE);
+
+ // initialize plan
+ RelDataTypeFactory typeFactory =
+ new JavaTypeFactoryImpl();
+ RelOptCluster cluster = RelOptCluster.create(volcanoPlanner, new RexBuilder(typeFactory));
+ RelBuilder builder = RelFactories.LOGICAL_BUILDER.create(cluster, null);
+ RelNode initRel = customPlanFunction.apply(builder);
+
+ // convert to HyperGraph
+ HepProgram program = new HepProgramBuilder()
+ .addMatchOrder(HepMatchOrder.BOTTOM_UP)
+ .addRuleInstance(CoreRules.JOIN_TO_HYPER_GRAPH)
+ .build();
+ HepPlanner hepPlanner = new HepPlanner(program);
+ hepPlanner.setRoot(initRel);
+ RelNode hyperGraph = hepPlanner.findBestExp();
+ assert hyperGraph instanceof HyperGraph : "The root node must be a HyperGraph, "
+ + "please check the custom plan function. The root node now is:\n"
+ + RelOptUtil.toString(hyperGraph);
+
+ // start dphyp enumeration
+ DphypForTest dphyp =
+ new DphypForTest(
+ (HyperGraph) hyperGraph,
+ builder,
+ hyperGraph.getCluster().getMetadataQuery(),
+ 127);
+ dphyp.startEnumerateJoin();
+
+ // verify the execution results of each candidate plan enumerated by dphyp
+ for (RelNode candidatePlan : dphyp.candiateLists) {
+ for (RelOptRule enumerableRule : EnumerableRules.rules()) {
+ volcanoPlanner.addRule(enumerableRule);
+ }
+ for (RelOptRule calcRule : RelOptRules.CALC_RULES) {
+ volcanoPlanner.addRule(calcRule);
+ }
+ volcanoPlanner.removeRule(EnumerableRules.ENUMERABLE_PROJECT_RULE);
+
+ candidatePlan =
Review Comment:
This part looks like a very useful utility function that could be factored out for other people to use: given a plan of a constant query (no tables), evaluate it.
##########
core/src/test/java/org/apache/calcite/test/DphypJoinReorderTest.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.test;
+
+import org.apache.calcite.DataContexts;
+import org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.calcite.adapter.enumerable.EnumerableInterpretable;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRules;
+import org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRules;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.hep.HepMatchOrder;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.plan.hep.HepProgramBuilder;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.rules.DpHyp;
+import org.apache.calcite.rel.rules.HyperGraph;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.runtime.Bindable;
+import org.apache.calcite.tools.RelBuilder;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+public class DphypJoinReorderTest {
+
+ @Test void testInnerLeftSemiJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // left join t3 on id2=id3
+ // where exists (select * from t4 where id3=id4)
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.LEFT,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 1, "Tom", 3, "Li")
+ .semiJoin(
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult = "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testChainInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id2=id3
+ // inner join t4 on id3=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, null, 3, "Li")
+ .values(new String[]{"id2", "name2"}, 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=2; name1=null; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testStarInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id1=id3
+ // inner join t4 on id1=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy", 1, "Tom")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom; id4=1; name4=Tom\n"
+ + "id1=2; name1=Lucy; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ /**
+ * Run the customized plan and check whether its execution results meet expectations (ignore row order).
+ *
+ * @param customPlanFunction a function that accepts one RelBuilder and produces a RelNode
+ * @param expectedResult expected result
+ */
+ void run(Function<RelBuilder, RelNode> customPlanFunction, String expectedResult) {
+ VolcanoPlanner volcanoPlanner = new VolcanoPlanner();
+ volcanoPlanner.addRelTraitDef(ConventionTraitDef.INSTANCE);
+ volcanoPlanner.addRelTraitDef(RelCollationTraitDef.INSTANCE);
+
+ // initialize plan
+ RelDataTypeFactory typeFactory =
+ new JavaTypeFactoryImpl();
+ RelOptCluster cluster = RelOptCluster.create(volcanoPlanner, new RexBuilder(typeFactory));
+ RelBuilder builder = RelFactories.LOGICAL_BUILDER.create(cluster, null);
+ RelNode initRel = customPlanFunction.apply(builder);
+
+ // convert to HyperGraph
+ HepProgram program = new HepProgramBuilder()
+ .addMatchOrder(HepMatchOrder.BOTTOM_UP)
+ .addRuleInstance(CoreRules.JOIN_TO_HYPER_GRAPH)
+ .build();
+ HepPlanner hepPlanner = new HepPlanner(program);
+ hepPlanner.setRoot(initRel);
+ RelNode hyperGraph = hepPlanner.findBestExp();
+ assert hyperGraph instanceof HyperGraph : "The root node must be a HyperGraph, "
+ + "please check the custom plan function. The root node now is:\n"
+ + RelOptUtil.toString(hyperGraph);
+
+ // start dphyp enumeration
+ DphypForTest dphyp =
+ new DphypForTest(
+ (HyperGraph) hyperGraph,
+ builder,
+ hyperGraph.getCluster().getMetadataQuery(),
+ 127);
+ dphyp.startEnumerateJoin();
+
+ // verify the execution results of each candidate plan enumerated by dphyp
+ for (RelNode candidatePlan : dphyp.candiateLists) {
+ for (RelOptRule enumerableRule : EnumerableRules.rules()) {
+ volcanoPlanner.addRule(enumerableRule);
+ }
+ for (RelOptRule calcRule : RelOptRules.CALC_RULES) {
+ volcanoPlanner.addRule(calcRule);
+ }
+ volcanoPlanner.removeRule(EnumerableRules.ENUMERABLE_PROJECT_RULE);
Review Comment:
what is special about this rule?
##########
core/src/test/java/org/apache/calcite/test/DphypJoinReorderTest.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.test;
+
+import org.apache.calcite.DataContexts;
+import org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.calcite.adapter.enumerable.EnumerableInterpretable;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRules;
+import org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRules;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.hep.HepMatchOrder;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.plan.hep.HepProgramBuilder;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.rules.DpHyp;
+import org.apache.calcite.rel.rules.HyperGraph;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.runtime.Bindable;
+import org.apache.calcite.tools.RelBuilder;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+public class DphypJoinReorderTest {
+
+ @Test void testInnerLeftSemiJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // left join t3 on id2=id3
+ // where exists (select * from t4 where id3=id4)
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.LEFT,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 1, "Tom", 3, "Li")
+ .semiJoin(
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult = "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testChainInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id2=id3
+ // inner join t4 on id3=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, null, 3, "Li")
+ .values(new String[]{"id2", "name2"}, 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=2; name1=null; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testStarInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id1=id3
+ // inner join t4 on id1=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy", 1, "Tom")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom; id4=1; name4=Tom\n"
+ + "id1=2; name1=Lucy; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ /**
+ * Run the customized plan and check whether its execution results meet expectations (ignore row order).
+ *
+ * @param customPlanFunction a function that accepts one RelBuilder and produces a RelNode
+ * @param expectedResult expected result
+ */
+ void run(Function<RelBuilder, RelNode> customPlanFunction, String expectedResult) {
+ VolcanoPlanner volcanoPlanner = new VolcanoPlanner();
+ volcanoPlanner.addRelTraitDef(ConventionTraitDef.INSTANCE);
+ volcanoPlanner.addRelTraitDef(RelCollationTraitDef.INSTANCE);
+
+ // initialize plan
+ RelDataTypeFactory typeFactory =
+ new JavaTypeFactoryImpl();
+ RelOptCluster cluster = RelOptCluster.create(volcanoPlanner, new RexBuilder(typeFactory));
+ RelBuilder builder = RelFactories.LOGICAL_BUILDER.create(cluster, null);
+ RelNode initRel = customPlanFunction.apply(builder);
+
+ // convert to HyperGraph
+ HepProgram program = new HepProgramBuilder()
+ .addMatchOrder(HepMatchOrder.BOTTOM_UP)
+ .addRuleInstance(CoreRules.JOIN_TO_HYPER_GRAPH)
+ .build();
+ HepPlanner hepPlanner = new HepPlanner(program);
+ hepPlanner.setRoot(initRel);
+ RelNode hyperGraph = hepPlanner.findBestExp();
+ assert hyperGraph instanceof HyperGraph : "The root node must be a HyperGraph, "
+ + "please check the custom plan function. The root node now is:\n"
+ + RelOptUtil.toString(hyperGraph);
+
+ // start dphyp enumeration
+ DphypForTest dphyp =
+ new DphypForTest(
+ (HyperGraph) hyperGraph,
+ builder,
+ hyperGraph.getCluster().getMetadataQuery(),
+ 127);
+ dphyp.startEnumerateJoin();
+
+ // verify the execution results of each candidate plan enumerated by dphyp
+ for (RelNode candidatePlan : dphyp.candiateLists) {
+ for (RelOptRule enumerableRule : EnumerableRules.rules()) {
+ volcanoPlanner.addRule(enumerableRule);
+ }
+ for (RelOptRule calcRule : RelOptRules.CALC_RULES) {
+ volcanoPlanner.addRule(calcRule);
+ }
+ volcanoPlanner.removeRule(EnumerableRules.ENUMERABLE_PROJECT_RULE);
+
+ candidatePlan =
+ volcanoPlanner.changeTraits(
+ candidatePlan,
+ cluster.traitSet().replace(EnumerableConvention.INSTANCE));
+ volcanoPlanner.setRoot(candidatePlan);
+ RelNode physicalPlan = volcanoPlanner.findBestExp();
+ volcanoPlanner.clear();
+
+ Bindable bindable =
+ EnumerableInterpretable.toBindable(
+ Collections.emptyMap(),
+ CalcitePrepare.Dummy.getSparkHandler(false),
+ (EnumerableRel) physicalPlan,
+ EnumerableRel.Prefer.ARRAY);
+ Enumerable<Object[]> result = bindable.bind(DataContexts.EMPTY);
+ Set<String> realResult =
+ formatResult(result.enumerator(), physicalPlan.getRowType().getFieldNames());
+
+ Set<String> expectedSet = new HashSet<>();
+ for (String row : expectedResult.split("\n")) {
+ expectedSet.add(row.trim());
+ }
+ assert realResult.equals(expectedSet) : "The result does not match the expected result.\n"
+ + "Expected: " + expectedSet
+ + "Actual: " + realResult;
+ }
+ }
+
+ Set<String> formatResult(Enumerator<Object[]> enumerator, List<String> fieldNames) {
+ Set<String> rowSet = new HashSet<>();
+ while (enumerator.moveNext()) {
+ StringBuilder output = new StringBuilder();
+ Object[] row = enumerator.current();
+ assert row.length == fieldNames.size() : "Result row length does not match field names size";
+ for (int i = 0; i < fieldNames.size(); i++) {
+ output.append(fieldNames.get(i))
+ .append("=")
+ .append(row[i] != null ? row[i].toString() : "null");
+
+ if (i < fieldNames.size() - 1) {
+ output.append("; ");
+ }
+ }
+ rowSet.add(output.toString());
+ }
+ return rowSet;
+ }
+
+ /** A test class for {@link DpHyp} that collects candidate plans. */
+ class DphypForTest extends DpHyp {
+
+ // a list for candidate plan that include all tables in the hypergraph
+ final List<RelNode> candiateLists;
Review Comment:
candidate is misspelled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2151918742
##########
core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java:
##########
@@ -10785,4 +10785,61 @@ private void checkLoptOptimizeJoinRule(LoptOptimizeJoinRule rule) {
.withRule(CoreRules.HYPER_GRAPH_OPTIMIZE, CoreRules.PROJECT_REMOVE, CoreRules.PROJECT_MERGE)
.check();
}
+
+ @Test void testVariousJoinTypeForDphyp() {
+ HepProgram program = new HepProgramBuilder()
+ .addMatchOrder(HepMatchOrder.BOTTOM_UP)
+ .addRuleInstance(CoreRules.PROJECT_MERGE)
+ .addRuleInstance(CoreRules.PROJECT_TO_SEMI_JOIN)
+ .addRuleInstance(CoreRules.JOIN_TO_HYPER_GRAPH)
+ .build();
+
+ sql("select emp.empno from "
+ + "emp_address inner join emp on emp_address.empno = emp.empno "
+ + "left join dept on emp.deptno = dept.deptno "
+ + "where exists(select * from dept_nested where dept.deptno = dept_nested.deptno)")
+ .withExpand(true)
+ .withDecorrelate(true)
+ .withPre(program)
+ .withRule(CoreRules.HYPER_GRAPH_OPTIMIZE, CoreRules.PROJECT_REMOVE, CoreRules.PROJECT_MERGE)
+ .check();
+ }
+
+ @Test void testVariousJoinTypeForDphyp2() {
Review Comment:
Maybe use join type (testLeftInnerSemiTypeForDphyp) as the name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2165710631
##########
core/src/test/java/org/apache/calcite/test/DphypJoinReorderTest.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.test;
+
+import org.apache.calcite.DataContexts;
+import org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.calcite.adapter.enumerable.EnumerableInterpretable;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRules;
+import org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRules;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.hep.HepMatchOrder;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.plan.hep.HepProgramBuilder;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.rules.DpHyp;
+import org.apache.calcite.rel.rules.HyperGraph;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.runtime.Bindable;
+import org.apache.calcite.tools.RelBuilder;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+public class DphypJoinReorderTest {
+
+ @Test void testInnerLeftSemiJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // left join t3 on id2=id3
+ // where exists (select * from t4 where id3=id4)
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.LEFT,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 1, "Tom", 3, "Li")
+ .semiJoin(
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult = "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testChainInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id2=id3
+ // inner join t4 on id3=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, null, 3, "Li")
+ .values(new String[]{"id2", "name2"}, 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=2; name1=null; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testStarInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id1=id3
+ // inner join t4 on id1=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy", 1, "Tom")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom; id4=1; name4=Tom\n"
+ + "id1=2; name1=Lucy; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ /**
+ * Run the customized plan and check whether its execution results meet expectations (ignore row order).
+ *
+ * @param customPlanFunction a function that accepts one RelBuilder and produces a RelNode
+ * @param expectedResult expected result
+ */
+ void run(Function<RelBuilder, RelNode> customPlanFunction, String expectedResult) {
+ VolcanoPlanner volcanoPlanner = new VolcanoPlanner();
+ volcanoPlanner.addRelTraitDef(ConventionTraitDef.INSTANCE);
+ volcanoPlanner.addRelTraitDef(RelCollationTraitDef.INSTANCE);
+
+ // initialize plan
+ RelDataTypeFactory typeFactory =
+ new JavaTypeFactoryImpl();
+ RelOptCluster cluster = RelOptCluster.create(volcanoPlanner, new RexBuilder(typeFactory));
+ RelBuilder builder = RelFactories.LOGICAL_BUILDER.create(cluster, null);
+ RelNode initRel = customPlanFunction.apply(builder);
+
+ // convert to HyperGraph
+ HepProgram program = new HepProgramBuilder()
+ .addMatchOrder(HepMatchOrder.BOTTOM_UP)
+ .addRuleInstance(CoreRules.JOIN_TO_HYPER_GRAPH)
+ .build();
+ HepPlanner hepPlanner = new HepPlanner(program);
+ hepPlanner.setRoot(initRel);
+ RelNode hyperGraph = hepPlanner.findBestExp();
+ assert hyperGraph instanceof HyperGraph : "The root node must be a HyperGraph, "
+ + "please check the custom plan function. The root node now is:\n"
+ + RelOptUtil.toString(hyperGraph);
+
+ // start dphyp enumeration
+ DphypForTest dphyp =
+ new DphypForTest(
+ (HyperGraph) hyperGraph,
+ builder,
+ hyperGraph.getCluster().getMetadataQuery(),
+ 127);
+ dphyp.startEnumerateJoin();
+
+ // verify the execution results of each candidate plan enumerated by dphyp
+ for (RelNode candidatePlan : dphyp.candiateLists) {
+ for (RelOptRule enumerableRule : EnumerableRules.rules()) {
+ volcanoPlanner.addRule(enumerableRule);
+ }
+ for (RelOptRule calcRule : RelOptRules.CALC_RULES) {
+ volcanoPlanner.addRule(calcRule);
+ }
+ volcanoPlanner.removeRule(EnumerableRules.ENUMERABLE_PROJECT_RULE);
Review Comment:
EnumerableProject does not implement the `implement` function. If you want to execute the plan, it seems that you must replace Project with Calc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2155942342
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -173,42 +202,140 @@ private void enumerateCmpRec(long csg, long cmp, long forbidden) {
private void emitCsgCmp(long csg, long cmp, List<HyperEdge> edges) {
RelNode child1 = dpTable.get(csg);
RelNode child2 = dpTable.get(cmp);
- if (child1 == null || child2 == null) {
- throw new IllegalArgumentException(
- "csg and cmp were not enumerated in the previous dp process");
- }
+ ImmutableList<HyperGraph.NodeState> csgOrder = resultInputOrder.get(csg);
+ ImmutableList<HyperGraph.NodeState> cmpOrder = resultInputOrder.get(cmp);
+ assert child1 != null && child2 != null && csgOrder != null && cmpOrder != null;
+ assert Long.bitCount(csg) == csgOrder.size() && Long.bitCount(cmp) == cmpOrder.size();
JoinRelType joinType = hyperGraph.extractJoinType(edges);
if (joinType == null) {
return;
}
- RexNode joinCond1 = hyperGraph.extractJoinCond(child1, child2, edges);
+ // verify whether the subgraph is legal by using the conflict rules in hyperedges
+ if (!hyperGraph.applicable(csg | cmp, edges)) {
+ return;
+ }
+
+ List<HyperGraph.NodeState> unionOrder = new ArrayList<>(csgOrder);
+ unionOrder.addAll(cmpOrder);
+ // build join condition from hyperedges. e.g.
+ // case.1
+ // csg: node0_projected [field0, field1], node1_projected [field0, field1],
+ //
+ // join
+ // / \
+ // node0 node1
+ //
+ // cmp: node2_projected [field0, field1]
+ // hyperedge1: node0.field0 = node2.field0
+ // hyperedge2: node1.field1 = node2.field1
+ // we will get join condition: ($0 = $4) and ($3 = $5)
+ //
+ // new_join(condition=[AND(($0 = $4), ($3 = $5))])
+ // / \
+ // join node2
+ // / \
+ // node0 node1
+ //
+ // case.2
+ // csg: node0_projected [field0, field1], node1_not_projected [field0, field1],
+ //
+ // join(joinType=semi/anti)
+ // / \
+ // node0 node1
+ //
+ // cmp: node2_projected [field0, field1]
+ // hyperedge1: node0.field0 = node2.field0
+ // hyperedge2: node0.field1 = node2.field1
+ // we will get join condition: ($0 = $2) and ($1 = $3)
+ //
+ // new_join(condition=[AND(($0 = $2), ($1 = $3))])
+ // / \
+ // join(joinType=semi/anti) node2
+ // / \
+ // node0 node1
+ RexNode joinCond1 = hyperGraph.extractJoinCond(unionOrder, csgOrder.size(), edges, joinType);
RelNode newPlan1 = builder
.push(child1)
.push(child2)
.join(joinType, joinCond1)
.build();
+ RelNode winPlan = newPlan1;
+ ImmutableList<HyperGraph.NodeState> winOrder = ImmutableList.copyOf(unionOrder);
+ assert verifyDpResultRowType(newPlan1, unionOrder);
Review Comment:
> I had suggested a test that validates all candidate hypergraphs, even the non-optimal ones, to improve the coverage of the code. I didn't see one. Is it possible to add one?
I added asserts during the enumeration process to verify the rowType of each candidate result. Is this OK? @mihaibudiu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2137477211
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,104 +75,160 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new AssertionError("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new AssertionError("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new AssertionError("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = initialLeftNodeBits;
+ rightEndpoint = initialRightNodeBits;
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
Review Comment:
https://github.com/apache/calcite/blob/eb04f942dd9102f2e6e7670db23c0ad97587e362/core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java#L66
Here, only the inner join condition is split by conjunction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2112390847
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -37,7 +40,12 @@ public class DpHyp {
private final HyperGraph hyperGraph;
- private final HashMap<Long, RelNode> dpTable;
+ private final Map<Long, RelNode> dpTable;
+
+ // record the node order corresponding to the best subgraph, which is used to convert
+ // the RexNodeAndFieldIndex in hyperedge to the RexInputRef in join condition, and permute
+ // final result
+ private final Map<Long, ImmutableList<Integer>> resultInputOrder;
Review Comment:
is the size the one shown in Tables 4/5 column 2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2099722877
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * On the correct and complete enumeration of the core search space.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
Review Comment:
Thank you for your suggestion. When reviewers think this PR is OK, I will create a new jira for this TODO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2918780234
https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/opt/xform/join_order_builder_test.go
@silundong
maybe can ref to cockroachdb's test cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2115113122
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
Review Comment:
Got it, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2131862257
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -1526,6 +1526,33 @@ LogicalJoin(condition=[=($1, $12)], joinType=[semi])
LogicalTableScan(table=[[scott, DEPT]])
LogicalTableScan(table=[[scott, EMP]])
LogicalTableScan(table=[[scott, BONUS]])
+]]>
+ </Resource>
+ </TestCase>
+ <TestCase name="testCartesianProductForDphyp">
+ <Resource name="sql">
+ <![CDATA[select emp.empno from emp_address, emp, emp_b, dept where emp_address.empno = emp.empno and dept.deptno = emp.deptno]]>
+ </Resource>
+ <Resource name="planBefore">
+ <![CDATA[
+LogicalProject(EMPNO=[$3])
+ HyperGraph(edges=[{0}——[INNER, =(vertex(0)_field(0), vertex(1)_field(0))]——{1},{0, 1}——[INNER, true]——{2},{1}——[INNER, =(vertex(3)_field(0), vertex(1)_field(7))]——{3}])
Review Comment:
maybe can add a refmap, key:table id, value: table name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2156068956
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -173,42 +202,140 @@ private void enumerateCmpRec(long csg, long cmp, long forbidden) {
private void emitCsgCmp(long csg, long cmp, List<HyperEdge> edges) {
RelNode child1 = dpTable.get(csg);
RelNode child2 = dpTable.get(cmp);
- if (child1 == null || child2 == null) {
- throw new IllegalArgumentException(
- "csg and cmp were not enumerated in the previous dp process");
- }
+ ImmutableList<HyperGraph.NodeState> csgOrder = resultInputOrder.get(csg);
+ ImmutableList<HyperGraph.NodeState> cmpOrder = resultInputOrder.get(cmp);
+ assert child1 != null && child2 != null && csgOrder != null && cmpOrder != null;
+ assert Long.bitCount(csg) == csgOrder.size() && Long.bitCount(cmp) == cmpOrder.size();
JoinRelType joinType = hyperGraph.extractJoinType(edges);
if (joinType == null) {
return;
}
- RexNode joinCond1 = hyperGraph.extractJoinCond(child1, child2, edges);
+ // verify whether the subgraph is legal by using the conflict rules in hyperedges
+ if (!hyperGraph.applicable(csg | cmp, edges)) {
+ return;
+ }
+
+ List<HyperGraph.NodeState> unionOrder = new ArrayList<>(csgOrder);
+ unionOrder.addAll(cmpOrder);
+ // build join condition from hyperedges. e.g.
+ // case.1
+ // csg: node0_projected [field0, field1], node1_projected [field0, field1],
+ //
+ // join
+ // / \
+ // node0 node1
+ //
+ // cmp: node2_projected [field0, field1]
+ // hyperedge1: node0.field0 = node2.field0
+ // hyperedge2: node1.field1 = node2.field1
+ // we will get join condition: ($0 = $4) and ($3 = $5)
+ //
+ // new_join(condition=[AND(($0 = $4), ($3 = $5))])
+ // / \
+ // join node2
+ // / \
+ // node0 node1
+ //
+ // case.2
+ // csg: node0_projected [field0, field1], node1_not_projected [field0, field1],
+ //
+ // join(joinType=semi/anti)
+ // / \
+ // node0 node1
+ //
+ // cmp: node2_projected [field0, field1]
+ // hyperedge1: node0.field0 = node2.field0
+ // hyperedge2: node0.field1 = node2.field1
+ // we will get join condition: ($0 = $2) and ($1 = $3)
+ //
+ // new_join(condition=[AND(($0 = $2), ($1 = $3))])
+ // / \
+ // join(joinType=semi/anti) node2
+ // / \
+ // node0 node1
+ RexNode joinCond1 = hyperGraph.extractJoinCond(unionOrder, csgOrder.size(), edges, joinType);
RelNode newPlan1 = builder
.push(child1)
.push(child2)
.join(joinType, joinCond1)
.build();
+ RelNode winPlan = newPlan1;
+ ImmutableList<HyperGraph.NodeState> winOrder = ImmutableList.copyOf(unionOrder);
+ assert verifyDpResultRowType(newPlan1, unionOrder);
Review Comment:
Got it. I will add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2155842422
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperEdge.java:
##########
@@ -73,10 +123,224 @@ public RexNode getCondition() {
@Override public String toString() {
StringBuilder sb = new StringBuilder();
- sb.append(LongBitmap.printBitmap(leftNodeBits))
+ sb.append(LongBitmap.printBitmap(getLeftEndpoint()))
.append("——[").append(joinType).append(", ").append(condition).append("]——")
- .append(LongBitmap.printBitmap(rightNodeBits));
+ .append(LongBitmap.printBitmap(getRightEndpoint()));
return sb.toString();
}
+ // It is convenient for the accept method in the HyperGraph class to call
+ public HyperEdge accept(RexShuttle shuttle) {
+ RexNode shuttleCondition = condition.accept(shuttle);
+ return new HyperEdge(
+ tes,
+ conflictRules,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ initialLeftNodeBits,
+ initialRightNodeBits,
+ joinType,
+ shuttleCondition);
+ }
+
+ /**
+ * When {@link JoinToHyperGraphRule} constructs a hypergraph , if the right child of Join is a
+ * HyperGraph class, all bitmaps related to inputs in this right child need to be shifted
+ * according to the number of inputs in the left child of Join.
+ *
+ * @param nodeOffset number of inputs in the left child
+ * @return HyperEdge after shifted
+ */
+ public HyperEdge adjustNodeBit(int nodeOffset) {
+ RexShuttle shiftNodeIndexShuttle = new RexShuttle() {
+ @Override public RexNode visitNodeAndFieldIndex(RexNodeAndFieldIndex nodeAndFieldIndex) {
+ return new RexNodeAndFieldIndex(
+ nodeAndFieldIndex.getNodeIndex() << nodeOffset,
+ nodeAndFieldIndex.getFieldIndex(),
+ nodeAndFieldIndex.getName(),
+ nodeAndFieldIndex.getType());
+ }
+ };
+ RexNode shiftedCondition = condition.accept(shiftNodeIndexShuttle);
+ List<ConflictRule> shiftedConflictRules = new ArrayList<>();
+ for (ConflictRule conflictRule : conflictRules) {
+ shiftedConflictRules.add(conflictRule.shift(nodeOffset));
+ }
+ return new HyperEdge(
+ tes.shift(nodeOffset),
+ shiftedConflictRules,
+ leftNodeUsedInPredicate << nodeOffset,
+ rightNodeUsedInPredicate << nodeOffset,
+ initialLeftNodeBits << nodeOffset,
+ initialRightNodeBits << nodeOffset,
+ joinType,
+ shiftedCondition);
+ }
+
+ /**
+ * Build hyperedges corresponding to a join operator. The construction of a HyperEdge includes
+ * total eligibility set, conflict rules, node-related bitmaps, predicate expressions (RexInputRef
+ * is replaced by RexNodeAndFieldIndex) and join type.
+ *
+ * @param inputRefToNodeIndexMap a map from inputRef to node index
+ * @param inputRefToFieldIndexMap a map from inputRef to field index
+ * @param joinConds join conditions
+ * @param conflictRules all conflict rules
+ * @param joinType join type
+ * @param leftNodeCount number of inputs in the left child of Join
+ * @param nodeCount number of inputs of Join
+ * @return HyperEdge list
+ */
+ public static List<HyperEdge> createHyperEdgesFromJoinConds(
+ Map<Integer, Integer> inputRefToNodeIndexMap,
+ Map<Integer, Integer> inputRefToFieldIndexMap,
+ List<RexNode> joinConds,
+ List<ConflictRule> conflictRules,
+ JoinRelType joinType,
+ int leftNodeCount,
+ int nodeCount) {
+ // for join operator o, generate bitmaps of the relations on the left/right sides
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, nodeCount);
+
+ List<HyperEdge> edges = new ArrayList<>();
+ for (RexNode joinCond : joinConds) {
+ List<Integer> leftRefs = new ArrayList<>();
+ List<Integer> rightRefs = new ArrayList<>();
+ RexShuttle inputRef2NodeAndFieldIndexShuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
+ Integer nodeIndex = inputRefToNodeIndexMap.get(inputRef.getIndex());
+ assert nodeIndex != null;
+ // collect left input index and right input index
+ if (nodeIndex < leftNodeCount) {
+ leftRefs.add(nodeIndex);
+ } else {
+ rightRefs.add(nodeIndex);
+ }
+ Integer fieldIndex = inputRefToFieldIndexMap.get(inputRef.getIndex());
+ assert fieldIndex != null;
+
+ return new RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
+ }
+ };
+ RexNode hyperEdgeCondition = joinCond.accept(inputRef2NodeAndFieldIndexShuttle);
+
+ long leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ long rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
+ TotalEligibilitySet tes;
+ List<ConflictRule> conflictRulesAfterAbsorb = new ArrayList<>();
+ if (leftNodeUsedInPredicate == 0 || rightNodeUsedInPredicate == 0) {
+ // when dealing with cross products or degenerate predicates, construct total eligibility
+ // set using all nodes on the left and right sides (as the endpoint of the hyperedge). Cross
+ // products and degenerate predicates should be rare in real queries, this behavior
Review Comment:
Got it. I will update the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2986406394
> I had suggested a test that validates all candidate hypergraphs, even the non-optimal ones, to improve the coverage of the code. I didn't see one. Is it possible to add one?
https://github.com/apache/calcite/blob/0c238dcb0229d6797bf2818ab08cf9ed766c234c/core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java#L360
I added assert during the enumeration process to verify the rowType of each candidate result. Is this ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2155836461
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictRule.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+/**
+ * A conflict rule (CR) is a pair of table sets denoted by T1 → T2, which means that if T1
+ * intersects with the table under the join operator, T2 must be included in this join. With every
+ * hyperedge in hypergraph, we associate a set of conflict rules.
+ *
+ * @see HyperEdge
+ * @see ConflictDetectionHelper
+ */
+public class ConflictRule {
+
Review Comment:
I think it may emphasize the meaning of **node set** more. What do you think of _this is a bitmap where indexes refer to input of hypergraph_?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2111461491
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperEdge.java:
##########
@@ -20,42 +20,96 @@
import org.apache.calcite.rel.core.JoinRelType;
import org.apache.calcite.rex.RexNode;
+import java.util.HashMap;
+import java.util.Map;
+
/**
* Edge in HyperGraph, that represents a join predicate.
*/
@Experimental
public class HyperEdge {
- private final long leftNodeBits;
+ // equivalent to the l-tes in CD-C paper
+ private final long leftEndpoint;
Review Comment:
As you said, I prefer to use long.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2157672652
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -173,42 +202,140 @@ private void enumerateCmpRec(long csg, long cmp, long forbidden) {
private void emitCsgCmp(long csg, long cmp, List<HyperEdge> edges) {
RelNode child1 = dpTable.get(csg);
RelNode child2 = dpTable.get(cmp);
- if (child1 == null || child2 == null) {
- throw new IllegalArgumentException(
- "csg and cmp were not enumerated in the previous dp process");
- }
+ ImmutableList<HyperGraph.NodeState> csgOrder = resultInputOrder.get(csg);
+ ImmutableList<HyperGraph.NodeState> cmpOrder = resultInputOrder.get(cmp);
+ assert child1 != null && child2 != null && csgOrder != null && cmpOrder != null;
+ assert Long.bitCount(csg) == csgOrder.size() && Long.bitCount(cmp) == cmpOrder.size();
JoinRelType joinType = hyperGraph.extractJoinType(edges);
if (joinType == null) {
return;
}
- RexNode joinCond1 = hyperGraph.extractJoinCond(child1, child2, edges);
+ // verify whether the subgraph is legal by using the conflict rules in hyperedges
+ if (!hyperGraph.applicable(csg | cmp, edges)) {
+ return;
+ }
+
+ List<HyperGraph.NodeState> unionOrder = new ArrayList<>(csgOrder);
+ unionOrder.addAll(cmpOrder);
+ // build join condition from hyperedges. e.g.
+ // case.1
+ // csg: node0_projected [field0, field1], node1_projected [field0, field1],
+ //
+ // join
+ // / \
+ // node0 node1
+ //
+ // cmp: node2_projected [field0, field1]
+ // hyperedge1: node0.field0 = node2.field0
+ // hyperedge2: node1.field1 = node2.field1
+ // we will get join condition: ($0 = $4) and ($3 = $5)
+ //
+ // new_join(condition=[AND(($0 = $4), ($3 = $5))])
+ // / \
+ // join node2
+ // / \
+ // node0 node1
+ //
+ // case.2
+ // csg: node0_projected [field0, field1], node1_not_projected [field0, field1],
+ //
+ // join(joinType=semi/anti)
+ // / \
+ // node0 node1
+ //
+ // cmp: node2_projected [field0, field1]
+ // hyperedge1: node0.field0 = node2.field0
+ // hyperedge2: node0.field1 = node2.field1
+ // we will get join condition: ($0 = $2) and ($1 = $3)
+ //
+ // new_join(condition=[AND(($0 = $2), ($1 = $3))])
+ // / \
+ // join(joinType=semi/anti) node2
+ // / \
+ // node0 node1
+ RexNode joinCond1 = hyperGraph.extractJoinCond(unionOrder, csgOrder.size(), edges, joinType);
RelNode newPlan1 = builder
.push(child1)
.push(child2)
.join(joinType, joinCond1)
.build();
+ RelNode winPlan = newPlan1;
+ ImmutableList<HyperGraph.NodeState> winOrder = ImmutableList.copyOf(unionOrder);
+ assert verifyDpResultRowType(newPlan1, unionOrder);
Review Comment:
please ask for a new review when you have made this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2115750737
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,104 +75,160 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new AssertionError("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new AssertionError("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new AssertionError("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = initialLeftNodeBits;
+ rightEndpoint = initialRightNodeBits;
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
Review Comment:
https://github.com/cockroachdb/cockroach/blob/475172846823a877e07eabb5e8da1a996e510d17/pkg/sql/opt/xform/join_order_builder.go#L438
For any given non-inner join, exactly one edge is constructed.
do we have same logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "suibianwanwank (via GitHub)" <gi...@apache.org>.
suibianwanwank commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2154175490
##########
core/src/main/java/org/apache/calcite/rex/RexNodeAndFieldIndex.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rex;
+
+import org.apache.calcite.rel.type.RelDataType;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.Objects;
+
+/**
+ * RexNodeAndFieldIndex has the same meaning as {@link RexInputRef}, they are both reference a
+ * field of an input relational expression. The difference is that, RexNodeAndFieldIndex uses the
+ * input index and the relative position of field in input rowType.
+ *
+ * <p> For example, if the inputs to a join are
+ *
+ * <ul>
+ * <li>Input #0: EMP(EMPNO, ENAME, DEPTNO) and</li>
+ * <li>Input #1: DEPT(DEPTNO AS DEPTNO2, DNAME)</li>
+ * </ul>
+ *
+ * <p>then the fields are:
+ *
+ * <ul>
+ * <li>Node #0, Field #0: EMPNO</li>
+ * <li>Node #0, Field #1: ENAME</li>
+ * <li>Node #0, Field #2: DEPTNO (from EMP)</li>
+ * <li>Node #1, Field #0: DEPTNO2 (from DEPT)</li>
+ * <li>Node #1, Field #1: DNAME</li>
+ * </ul>
+ *
+ * <p> If in some cases, the inputs order of a relation is frequently adjusted and it is difficult
+ * to maintain the correct RexInputRef, you can consider temporarily replacing RexInputRef with
+ * RexNodeAndFieldIndex.
+ *
+ * @see org.apache.calcite.rel.rules.JoinToHyperGraphRule
+ * @see org.apache.calcite.rel.rules.HyperGraph#extractJoinCond
+ * @see org.apache.calcite.rel.rules.HyperEdge#createHyperEdgesFromJoinConds
+ */
+public class RexNodeAndFieldIndex extends RexVariable {
Review Comment:
This comment is really good for me!
And why name it RexNodeAndFieldIndex? It looks a bit strange. How about HyperGraphInputRef?
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -1643,6 +1670,33 @@ LogicalProject(EMPNO=[$9])
LogicalJoin(condition=[=($3, $0)], joinType=[inner])
LogicalTableScan(table=[[CATALOG, SALES, EMP_ADDRESS]])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+]]>
+ </Resource>
+ </TestCase>
+ <TestCase name="testComplexPredicateForDphyp">
+ <Resource name="sql">
+ <![CDATA[select emp.empno from emp, emp_b, dept, dept_nested where emp.deptno + emp_b.empno = dept.deptno + dept_nested.deptno and (emp.empno = emp_b.empno or emp.ename = emp_b.ename) and sqrt(dept.deptno + dept_nested.deptno) > 2]]>
+ </Resource>
+ <Resource name="planBefore">
+ <![CDATA[
+LogicalProject(EMPNO=[$0])
+ HyperGraph(edges=[{0, 1, 2}——[INNER, =(+(node(0)_field(7), node(1)_field(0)), +(node(2)_field(0), node(3)_field(0)))]——{3},{2}——[INNER, >(POWER(+(node(2)_field(0), node(3)_field(0)), 0.5:DECIMAL(2, 1)), CAST(2):DOUBLE NOT NULL)]——{3},{0, 1}——[INNER, true]——{2},{0}——[INNER, OR(=(node(0)_field(0), node(1)_field(0)), =(node(0)_field(1), node(1)_field(1)))]——{1}])
Review Comment:
The test results look good. Is there a way we can make the plan appear more concise?
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -55,119 +53,146 @@ protected JoinToHyperGraphRule(Config config) {
final Join origJoin = call.rel(0);
final RelNode left = call.rel(1);
final RelNode right = call.rel(2);
- if (origJoin.getJoinType() != JoinRelType.INNER) {
+ if (unSupportedJoinType(origJoin.getJoinType())) {
Review Comment:
I prefer using supportedJoinType, although the number of supported types far exceeds the unsupported ones. However, in the future when someone adds joinType, they might not consider this aspect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2155719489
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictRule.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+/**
+ * A conflict rule (CR) is a pair of table sets denoted by T1 → T2, which means that if T1
+ * intersects with the table under the join operator, T2 must be included in this join. With every
+ * hyperedge in hypergraph, we associate a set of conflict rules.
+ *
+ * @see HyperEdge
+ * @see ConflictDetectionHelper
+ */
+public class ConflictRule {
+
Review Comment:
I would add a comment that this is a bitmap where indexes refer to an implicit hypergraph.
(I assume this is right)
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictRule.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+/**
+ * A conflict rule (CR) is a pair of table sets denoted by T1 → T2, which means that if T1
+ * intersects with the table under the join operator, T2 must be included in this join. With every
+ * hyperedge in hypergraph, we associate a set of conflict rules.
+ *
+ * @see HyperEdge
+ * @see ConflictDetectionHelper
+ */
+public class ConflictRule {
+
Review Comment:
I would add a comment that this is a bitmap where indexes refer to an implicit hypergraph.
(I assume this is right)
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperEdge.java:
##########
@@ -73,10 +123,224 @@ public RexNode getCondition() {
@Override public String toString() {
StringBuilder sb = new StringBuilder();
- sb.append(LongBitmap.printBitmap(leftNodeBits))
+ sb.append(LongBitmap.printBitmap(getLeftEndpoint()))
.append("——[").append(joinType).append(", ").append(condition).append("]——")
- .append(LongBitmap.printBitmap(rightNodeBits));
+ .append(LongBitmap.printBitmap(getRightEndpoint()));
return sb.toString();
}
+ // It is convenient for the accept method in the HyperGraph class to call
+ public HyperEdge accept(RexShuttle shuttle) {
+ RexNode shuttleCondition = condition.accept(shuttle);
+ return new HyperEdge(
+ tes,
+ conflictRules,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ initialLeftNodeBits,
+ initialRightNodeBits,
+ joinType,
+ shuttleCondition);
+ }
+
+ /**
+ * When {@link JoinToHyperGraphRule} constructs a hypergraph , if the right child of Join is a
+ * HyperGraph class, all bitmaps related to inputs in this right child need to be shifted
+ * according to the number of inputs in the left child of Join.
+ *
+ * @param nodeOffset number of inputs in the left child
+ * @return HyperEdge after shifted
Review Comment:
hyperedge produced after shifting
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,106 +75,205 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ rightEndpoint = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
- edges,
+ notProjectInputs,
+ leftSubEdges,
origJoin.getRowType());
call.transformTo(result);
}
- private static HyperEdge adjustNodeBit(HyperEdge hyperEdge, int nodeOffset, int fieldOffset) {
- RexNode newCondition = RexUtil.shift(hyperEdge.getCondition(), fieldOffset);
+ private static HyperEdge adjustNodeBit(HyperEdge hyperEdge, int nodeOffset) {
+ RexShuttle shiftNodeIndexShuttle = new RexShuttle() {
+ @Override protected List<RexNode> visitList(
+ List<? extends RexNode> exprs,
+ boolean @Nullable [] update) {
+ ImmutableList.Builder<RexNode> clonedOperands = ImmutableList.builder();
+ for (RexNode operand : exprs) {
+ RexNode clonedOperand;
+ if (operand instanceof HyperGraph.RexNodeAndFieldIndex) {
+ clonedOperand =
+ new HyperGraph.RexNodeAndFieldIndex(
+ ((HyperGraph.RexNodeAndFieldIndex) operand).nodeIndex + nodeOffset,
+ ((HyperGraph.RexNodeAndFieldIndex) operand).fieldIndex,
+ ((HyperGraph.RexNodeAndFieldIndex) operand).getName(),
+ operand.getType());
+ } else {
+ clonedOperand = operand.accept(this);
+ }
+ if ((clonedOperand != operand) && (update != null)) {
+ update[0] = true;
+ }
+ clonedOperands.add(clonedOperand);
+ }
+ return clonedOperands.build();
+ }
+ };
+ RexNode newCondition = hyperEdge.getCondition().accept(shiftNodeIndexShuttle);
+ Map<Long, Long> newConflictRules = new HashMap<>();
+ for (Map.Entry<Long, Long> entry : hyperEdge.getConflictRules().entrySet()) {
+ newConflictRules.put(entry.getKey() << nodeOffset, entry.getValue() << nodeOffset);
+ }
return new HyperEdge(
- hyperEdge.getLeftNodeBitmap() << nodeOffset,
- hyperEdge.getRightNodeBitmap() << nodeOffset,
+ hyperEdge.getLeftEndpoint() << nodeOffset,
+ hyperEdge.getRightEndpoint() << nodeOffset,
+ hyperEdge.getLeftNodeUsedInPredicate() << nodeOffset,
+ hyperEdge.getRightNodeUsedInPredicate() << nodeOffset,
+ newConflictRules,
+ hyperEdge.getInitialLeftNodeBits() << nodeOffset,
+ hyperEdge.getInitialRightNodeBits() << nodeOffset,
hyperEdge.getJoinType(),
newCondition);
}
+ private static boolean unSupportedJoinType(JoinRelType joinType) {
+ return joinType == JoinRelType.RIGHT
Review Comment:
Maybe in the future you will consider creating a preprocessing step to be run before this one which converts right joins into left joins. (Is there one already?)
##########
core/src/main/java/org/apache/calcite/rex/RexNodeAndFieldIndex.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rex;
+
+import org.apache.calcite.rel.type.RelDataType;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.Objects;
+
+/**
+ * RexNodeAndFieldIndex has the same meaning as {@link RexInputRef}, they are both reference a
+ * field of an input relational expression. The difference is that, RexNodeAndFieldIndex uses the
+ * input index and the relative position of field in input rowType.
+ *
+ * <p> For example, if the inputs to a join are
+ *
+ * <ul>
+ * <li>Input #0: EMP(EMPNO, ENAME, DEPTNO) and</li>
+ * <li>Input #1: DEPT(DEPTNO AS DEPTNO2, DNAME)</li>
+ * </ul>
+ *
+ * <p>then the fields are:
+ *
+ * <ul>
+ * <li>Node #0, Field #0: EMPNO</li>
+ * <li>Node #0, Field #1: ENAME</li>
+ * <li>Node #0, Field #2: DEPTNO (from EMP)</li>
+ * <li>Node #1, Field #0: DEPTNO2 (from DEPT)</li>
+ * <li>Node #1, Field #1: DNAME</li>
+ * </ul>
+ *
+ * <p> If in some cases, the inputs order of a relation is frequently adjusted and it is difficult
+ * to maintain the correct RexInputRef, you can consider temporarily replacing RexInputRef with
+ * RexNodeAndFieldIndex.
+ *
+ * @see org.apache.calcite.rel.rules.JoinToHyperGraphRule
+ * @see org.apache.calcite.rel.rules.HyperGraph#extractJoinCond
+ * @see org.apache.calcite.rel.rules.HyperEdge#createHyperEdgesFromJoinConds
+ */
+public class RexNodeAndFieldIndex extends RexVariable {
Review Comment:
Well, the class may prove useful outside of the hypergraph code.
It does not seem related to the hypergraph really.
But if this class is used in code outside this optimization pass it will probably need a lot of other implementations.
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperEdge.java:
##########
@@ -73,10 +123,224 @@ public RexNode getCondition() {
@Override public String toString() {
StringBuilder sb = new StringBuilder();
- sb.append(LongBitmap.printBitmap(leftNodeBits))
+ sb.append(LongBitmap.printBitmap(getLeftEndpoint()))
.append("——[").append(joinType).append(", ").append(condition).append("]——")
- .append(LongBitmap.printBitmap(rightNodeBits));
+ .append(LongBitmap.printBitmap(getRightEndpoint()));
return sb.toString();
}
+ // It is convenient for the accept method in the HyperGraph class to call
+ public HyperEdge accept(RexShuttle shuttle) {
+ RexNode shuttleCondition = condition.accept(shuttle);
+ return new HyperEdge(
+ tes,
+ conflictRules,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ initialLeftNodeBits,
+ initialRightNodeBits,
+ joinType,
+ shuttleCondition);
+ }
+
+ /**
+ * When {@link JoinToHyperGraphRule} constructs a hypergraph , if the right child of Join is a
+ * HyperGraph class, all bitmaps related to inputs in this right child need to be shifted
+ * according to the number of inputs in the left child of Join.
+ *
+ * @param nodeOffset number of inputs in the left child
+ * @return HyperEdge after shifted
+ */
+ public HyperEdge adjustNodeBit(int nodeOffset) {
+ RexShuttle shiftNodeIndexShuttle = new RexShuttle() {
+ @Override public RexNode visitNodeAndFieldIndex(RexNodeAndFieldIndex nodeAndFieldIndex) {
+ return new RexNodeAndFieldIndex(
+ nodeAndFieldIndex.getNodeIndex() << nodeOffset,
+ nodeAndFieldIndex.getFieldIndex(),
+ nodeAndFieldIndex.getName(),
+ nodeAndFieldIndex.getType());
+ }
+ };
+ RexNode shiftedCondition = condition.accept(shiftNodeIndexShuttle);
+ List<ConflictRule> shiftedConflictRules = new ArrayList<>();
+ for (ConflictRule conflictRule : conflictRules) {
+ shiftedConflictRules.add(conflictRule.shift(nodeOffset));
+ }
+ return new HyperEdge(
+ tes.shift(nodeOffset),
+ shiftedConflictRules,
+ leftNodeUsedInPredicate << nodeOffset,
+ rightNodeUsedInPredicate << nodeOffset,
+ initialLeftNodeBits << nodeOffset,
+ initialRightNodeBits << nodeOffset,
+ joinType,
+ shiftedCondition);
+ }
+
+ /**
+ * Build hyperedges corresponding to a join operator. The construction of a HyperEdge includes
+ * total eligibility set, conflict rules, node-related bitmaps, predicate expressions (RexInputRef
+ * is replaced by RexNodeAndFieldIndex) and join type.
+ *
+ * @param inputRefToNodeIndexMap a map from inputRef to node index
+ * @param inputRefToFieldIndexMap a map from inputRef to field index
+ * @param joinConds join conditions
+ * @param conflictRules all conflict rules
+ * @param joinType join type
+ * @param leftNodeCount number of inputs in the left child of Join
+ * @param nodeCount number of inputs of Join
+ * @return HyperEdge list
+ */
+ public static List<HyperEdge> createHyperEdgesFromJoinConds(
+ Map<Integer, Integer> inputRefToNodeIndexMap,
+ Map<Integer, Integer> inputRefToFieldIndexMap,
+ List<RexNode> joinConds,
+ List<ConflictRule> conflictRules,
+ JoinRelType joinType,
+ int leftNodeCount,
+ int nodeCount) {
+ // for join operator o, generate bitmaps of the relations on the left/right sides
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, nodeCount);
+
+ List<HyperEdge> edges = new ArrayList<>();
+ for (RexNode joinCond : joinConds) {
+ List<Integer> leftRefs = new ArrayList<>();
+ List<Integer> rightRefs = new ArrayList<>();
+ RexShuttle inputRef2NodeAndFieldIndexShuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
+ Integer nodeIndex = inputRefToNodeIndexMap.get(inputRef.getIndex());
+ assert nodeIndex != null;
+ // collect left input index and right input index
+ if (nodeIndex < leftNodeCount) {
+ leftRefs.add(nodeIndex);
+ } else {
+ rightRefs.add(nodeIndex);
+ }
+ Integer fieldIndex = inputRefToFieldIndexMap.get(inputRef.getIndex());
+ assert fieldIndex != null;
+
+ return new RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
+ }
+ };
+ RexNode hyperEdgeCondition = joinCond.accept(inputRef2NodeAndFieldIndexShuttle);
+
+ long leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ long rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
+ TotalEligibilitySet tes;
+ List<ConflictRule> conflictRulesAfterAbsorb = new ArrayList<>();
+ if (leftNodeUsedInPredicate == 0 || rightNodeUsedInPredicate == 0) {
+ // when dealing with cross products or degenerate predicates, construct total eligibility
+ // set using all nodes on the left and right sides (as the endpoint of the hyperedge). Cross
+ // products and degenerate predicates should be rare in real queries, this behavior
Review Comment:
People write many queries using cross-products (see TPC-H).
So I suspect that what this means is that in practice few real optimized queries will contain cross-products.
This is an important distinction, because you want to handle programs with cross-products.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2112385803
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperGraph.java:
##########
@@ -401,69 +453,67 @@ public RexNode extractJoinCond(RelNode left, RelNode right, List<HyperEdge> edge
};
for (HyperEdge edge : edges) {
- RexNode inputRefCond = edge.getCondition().accept(inputName2InputRefShuttle);
+ RexNode inputRefCond = edge.getCondition().accept(shuttle);
joinConds.add(inputRefCond);
}
- return RexUtil.composeConjunction(left.getCluster().getRexBuilder(), joinConds);
+ return RexUtil.composeConjunction(getCluster().getRexBuilder(), joinConds);
Review Comment:
are you saying that a predicate such as `sqrt(left.x + right.y) > 2` would be handled correctly?
or a predicate like `a.x = b.y OR a.x = b.z`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2116144373
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,104 +75,160 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new AssertionError("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new AssertionError("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new AssertionError("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = initialLeftNodeBits;
+ rightEndpoint = initialRightNodeBits;
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
Review Comment:
Not yet. I looked at the code you quoted, but I didn't see why it does this. Maybe I'm missing something. Could you please explain what the purpose of doing this is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2920200566
There are two levels: you can generate the plans and check that the output type is right; this will ensure that the plans are validated. This will catch some problems and should be easy to do.
If you can build some infra where the queries are also executed on a fixed small dataset, this would be extremely valuable, since then you can have end-to-end tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2098079459
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * On the correct and complete enumeration of the core search space.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
+ List<HyperEdge> leftSubEdges,
+ List<HyperEdge> rightSubEdges,
+ JoinRelType joinType) {
+ Map<Long, Long> conflictRules = new HashMap<>();
+ for (HyperEdge leftSubEdge : leftSubEdges) {
+ if (!isAssociative(leftSubEdge.getJoinType(), joinType)) {
+ if (leftSubEdge.getLeftNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ leftSubEdge.getInitialRightNodeBits(),
+ leftSubEdge.getLeftNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ leftSubEdge.getInitialRightNodeBits(),
+ leftSubEdge.getInitialLeftNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+
+ }
+ if (!isLeftAsscom(leftSubEdge.getJoinType(), joinType)) {
+ if (leftSubEdge.getRightNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ leftSubEdge.getInitialLeftNodeBits(),
+ leftSubEdge.getRightNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ leftSubEdge.getInitialLeftNodeBits(),
+ leftSubEdge.getInitialRightNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ }
+
+ for (HyperEdge rightSubEdge : rightSubEdges) {
+ if (!isAssociative(joinType, rightSubEdge.getJoinType())) {
+ if (rightSubEdge.getRightNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ rightSubEdge.getInitialLeftNodeBits(),
+ rightSubEdge.getRightNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ rightSubEdge.getInitialLeftNodeBits(),
+ rightSubEdge.getInitialRightNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ if (!isRightAsscom(joinType, rightSubEdge.getJoinType())) {
+ if (rightSubEdge.getLeftNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ rightSubEdge.getInitialRightNodeBits(),
+ rightSubEdge.getLeftNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ rightSubEdge.getInitialRightNodeBits(),
+ rightSubEdge.getInitialLeftNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ }
+ return conflictRules;
+ }
+
+ /**
+ * For conflict rule T1 -> T2, if T1 and tes have intersection, we can add T2 into tes and remove
+ * this rule. See section 5.5 in paper.
+ *
+ * @param tes tes
+ * @param conflictRulesAfterAbsorb conflict rules after absorbing
+ * @param initialconflictRules conflict rules before absorbing
+ * @return tes after absorbing conflict rules
+ */
+ public static long absorbConflictRulesIntoTES(
+ long tes,
+ Map<Long, Long> conflictRulesAfterAbsorb,
+ Map<Long, Long> initialconflictRules) {
+ for (Map.Entry<Long, Long> rule : initialconflictRules.entrySet()) {
+ if (LongBitmap.isOverlap(tes, rule.getKey())) {
+ tes |= rule.getValue();
+ continue;
+ }
+ conflictRulesAfterAbsorb.put(rule.getKey(), rule.getValue());
+ }
+ return tes;
+ }
+
+ /**
+ * Check whether the operator is applicable through the conflict rule.
+ *
+ * @param subGraph table set of the current join operator
+ * @param edges hyper edges with conflict rules
+ * @return true if the operator is applicable
+ */
+ public static boolean applicable(long subGraph, List<HyperEdge> edges) {
+ for (HyperEdge edge : edges) {
+ for (Map.Entry<Long, Long> rule : edge.getConflictRules().entrySet()) {
+ if (LongBitmap.isOverlap(subGraph, rule.getKey())
+ && !LongBitmap.isSubSet(rule.getValue(), subGraph)) {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ public static boolean isCommutative(JoinRelType operator) {
+ return operator == JoinRelType.INNER || operator == JoinRelType.FULL;
+ }
+
+ public static boolean isAssociative(JoinRelType operatorA, JoinRelType operatorB) {
+ Integer indexA = INDEX_OF_TABLE.get(operatorA);
Review Comment:
Yes, it is possible to abstract a method to represent the common part. But the common part is relatively small, and the common method needs to return a `Pair`, which I think becomes more complicated. In addition, I don't think similar methods will be expanded in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2103416928
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * On the correct and complete enumeration of the core search space.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
Review Comment:
I haven't read the paper, but for the case of null-rejecting predicates the outer joins can be rewritten into inner joins. Shouldn't we assume this has been done prior to this optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2111460183
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
Review Comment:
I think this method does not belong to the `HyperGraph` class. Specifically, this method is called when the `JoinToHyperGraphRule` rule matches a `Join`. When the left child or right child of this Join is a `HyperGraph`, there are elements in leftSubEdges or rightSubEdges, and the for loops in this method will be executed. At this time, the new `HyperGraph` is not formed.
Now this method only uses `JoinRelType`, why does it need a complete join node as a argument?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2915731419
> I haven't really checked that the algorithm is correctly implemented, I left only comments about the code structure. I think this needs way more tests, even if they are slow tests. You need for example to test on plans with lots of joins and check that all the orders generated are valid and generate the same result, even if they are not optimal. This will exercise the code much more deeply than the tests you have now.
Thank you very much for your detailed review. I have replied to some comments and hope to discuss further with you. I will modify the other comments that I did not reply to.
In addition, when you say more tests, you mean that I should add some test cases in `RelOptRulesTest`, which have many tables involved in join graph, although the enumeration process may be slow in this case. Am I understanding it correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "suibianwanwank (via GitHub)" <gi...@apache.org>.
suibianwanwank commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2137372206
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -1526,6 +1526,33 @@ LogicalJoin(condition=[=($1, $12)], joinType=[semi])
LogicalTableScan(table=[[scott, DEPT]])
LogicalTableScan(table=[[scott, EMP]])
LogicalTableScan(table=[[scott, BONUS]])
+]]>
+ </Resource>
+ </TestCase>
+ <TestCase name="testCartesianProductForDphyp">
+ <Resource name="sql">
+ <![CDATA[select emp.empno from emp_address, emp, emp_b, dept where emp_address.empno = emp.empno and dept.deptno = emp.deptno]]>
+ </Resource>
+ <Resource name="planBefore">
+ <![CDATA[
+LogicalProject(EMPNO=[$3])
+ HyperGraph(edges=[{0}——[INNER, =(vertex(0)_field(0), vertex(1)_field(0))]——{1},{0, 1}——[INNER, true]——{2},{1}——[INNER, =(vertex(3)_field(0), vertex(1)_field(7))]——{3}])
Review Comment:
Perhaps we need to add some ways to observe the DP process (could be hooks or verbose logs), similar to MySQL's DynamicTable log? When users apply this rule and want to adjust the cost model, it might function as a black box.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2139356313
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -1526,6 +1526,33 @@ LogicalJoin(condition=[=($1, $12)], joinType=[semi])
LogicalTableScan(table=[[scott, DEPT]])
LogicalTableScan(table=[[scott, EMP]])
LogicalTableScan(table=[[scott, BONUS]])
+]]>
+ </Resource>
+ </TestCase>
+ <TestCase name="testCartesianProductForDphyp">
+ <Resource name="sql">
+ <![CDATA[select emp.empno from emp_address, emp, emp_b, dept where emp_address.empno = emp.empno and dept.deptno = emp.deptno]]>
+ </Resource>
+ <Resource name="planBefore">
+ <![CDATA[
+LogicalProject(EMPNO=[$3])
+ HyperGraph(edges=[{0}——[INNER, =(vertex(0)_field(0), vertex(1)_field(0))]——{1},{0, 1}——[INNER, true]——{2},{1}——[INNER, =(vertex(3)_field(0), vertex(1)_field(7))]——{3}])
Review Comment:
How about adding a new Logger for dphyp enumeration in `CalciteTrace`.
We can log every legal csg-cmp pair during enumerating, including endpoints, conditions, join type, cost value, whether to update dpTable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-3003443931
Thanks all for the reviews and helpful suggestions !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2098009043
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -17826,16 +17911,16 @@ LogicalProject(EMPNO=[$0])
</Resource>
<Resource name="planAfter">
<![CDATA[
-LogicalProject(EMPNO=[$19])
- LogicalJoin(condition=[AND(=($19, $0), =(+($8, $0), +($17, $13)))], joinType=[inner])
- LogicalTableScan(table=[[CATALOG, SALES, EMP_ADDRESS]])
+LogicalProject(EMPNO=[$16])
Review Comment:
I think this is caused by double precision.
Before explaining this problem, I need to explain that in the previous pr, in order to conveniently represent `RexInputRef` during enumeration, a unique alias will be generated for each input before enumeration (that is, an additional Project operator will be generated).
However, relying on strings and input rowType will cause trouble when processing semi/anti joins (the right child will not be projected). Therefore, in this PR, I changed it to use node index and the relative position of the field in the node to represent `RexInputRef`.
So, even if the join order is essentially the same, the cost value of the plan in this pr and the previous pr is different during enumeration (the previous pr has an additional Project).
Specifically,
In the previous pr:
t1 join t2 cost=154.21564(with additional Project)
t2 join t1 cost=154.21564(with additional Project) **_winner_**
In this pr:
t1 join t2 cost=92.21564 **_winner_**
t2 join t1 cost=92.21564000000001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2101538165
##########
core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java:
##########
@@ -10720,4 +10720,52 @@ private void checkLoptOptimizeJoinRule(LoptOptimizeJoinRule rule) {
sql(sql).withRule(CoreRules.AGGREGATE_MIN_MAX_TO_LIMIT)
.check();
}
+
+ @Test void testOuterJoinForDphyp() {
Review Comment:
I am sorry to put forward my own opinion. I admire your algorithm transplantation ability and this algorithm may be very good, but referring to the standards of other computing engines, we may also need to introduce benchmarks or stress testing instances to determine how much performance improvement is obtained in what scenarios，the test cases do not include relevant benchmarks, so strictly speaking, it may not be possible to confirm that performance can be improved, so as to explain the benefits of this modification. @silundong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2131806769
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,104 +75,160 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new AssertionError("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new AssertionError("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new AssertionError("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = initialLeftNodeBits;
+ rightEndpoint = initialRightNodeBits;
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
Review Comment:
Thank you very much for pointing this out! This is indeed a bug in the current code. Inner join can be rewritten as cross product + where. Due to this property, inner join condition can be split by conjunction and freely combine with other inner join conditions. This privilege cannot be extended to non-inner joins, so we cannot split the condition of non-inner joins, that is, only generate one edge. I will update it in the next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2137018998
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -1526,6 +1526,33 @@ LogicalJoin(condition=[=($1, $12)], joinType=[semi])
LogicalTableScan(table=[[scott, DEPT]])
LogicalTableScan(table=[[scott, EMP]])
LogicalTableScan(table=[[scott, BONUS]])
+]]>
+ </Resource>
+ </TestCase>
+ <TestCase name="testCartesianProductForDphyp">
+ <Resource name="sql">
+ <![CDATA[select emp.empno from emp_address, emp, emp_b, dept where emp_address.empno = emp.empno and dept.deptno = emp.deptno]]>
+ </Resource>
+ <Resource name="planBefore">
+ <![CDATA[
+LogicalProject(EMPNO=[$3])
+ HyperGraph(edges=[{0}——[INNER, =(vertex(0)_field(0), vertex(1)_field(0))]——{1},{0, 1}——[INNER, true]——{2},{1}——[INNER, =(vertex(3)_field(0), vertex(1)_field(7))]——{3}])
Review Comment:
Do you mean the hyperedge print? I think that input doesn't necessarily contain just a table, may be it is a subplan with agg-join-(t1,t2) or union-(t1,t2...) etc. So maybe it would be more intuitive to print the input index?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2137383920
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,104 +75,160 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new AssertionError("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new AssertionError("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new AssertionError("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = initialLeftNodeBits;
+ rightEndpoint = initialRightNodeBits;
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
Review Comment:
hi, which part code can reflact 'For any given non-inner join, exactly one edge is constructed.'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2111460183
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
Review Comment:
I think this method does not belong to the `HyperGraph` class. Specifically, this method is called when the `JoinToHyperGraphRule` matches a `Join`. When the left child or right child of this Join is a `HyperGraph`, there are elements in leftSubEdges or rightSubEdges, and the for loops in this method will be executed. At this time, the new `HyperGraph` is not formed.
Now this method only uses `JoinRelType`, why does it need a complete join node as a argument?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2110407284
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
Review Comment:
the tables in the paper have 7 operators, but it looks like you are omitting cross-product and group join (first and last). I would add this in the comment above.
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
Review Comment:
this is table 2, the next 2 tables are from table 3 in the paper, why not be more explicit in the comment?
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
Review Comment:
does this method belong to the hypergraph?
it could take a join node as an argument.
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
+ List<HyperEdge> leftSubEdges,
+ List<HyperEdge> rightSubEdges,
+ JoinRelType joinType) {
+ Map<Long, Long> conflictRules = new HashMap<>();
+ for (HyperEdge leftSubEdge : leftSubEdges) {
+ if (!isAssociative(leftSubEdge.getJoinType(), joinType)) {
+ if (leftSubEdge.getLeftNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ leftSubEdge.getInitialRightNodeBits(),
+ leftSubEdge.getLeftNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ leftSubEdge.getInitialRightNodeBits(),
+ leftSubEdge.getInitialLeftNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+
+ }
+ if (!isLeftAsscom(leftSubEdge.getJoinType(), joinType)) {
+ if (leftSubEdge.getRightNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ leftSubEdge.getInitialLeftNodeBits(),
+ leftSubEdge.getRightNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ leftSubEdge.getInitialLeftNodeBits(),
+ leftSubEdge.getInitialRightNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ }
+
+ for (HyperEdge rightSubEdge : rightSubEdges) {
+ if (!isAssociative(joinType, rightSubEdge.getJoinType())) {
+ if (rightSubEdge.getRightNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ rightSubEdge.getInitialLeftNodeBits(),
+ rightSubEdge.getRightNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ rightSubEdge.getInitialLeftNodeBits(),
+ rightSubEdge.getInitialRightNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ if (!isRightAsscom(joinType, rightSubEdge.getJoinType())) {
+ if (rightSubEdge.getLeftNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ rightSubEdge.getInitialRightNodeBits(),
+ rightSubEdge.getLeftNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ rightSubEdge.getInitialRightNodeBits(),
+ rightSubEdge.getInitialLeftNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ }
+ return conflictRules;
+ }
+
+ /**
+ * For conflict rule <code>T1 -> T2</code>, if T1 and tes have intersection, we can add T2
+ * into tes and remove this rule. See section 5.5 in paper.
+ *
+ * @param tes tes
+ * @param conflictRulesAfterAbsorb conflict rules after absorbing
+ * @param initialconflictRules conflict rules before absorbing
+ * @return tes after absorbing conflict rules
+ */
+ public static long absorbConflictRulesIntoTES(
+ long tes,
+ Map<Long, Long> conflictRulesAfterAbsorb,
+ Map<Long, Long> initialconflictRules) {
+ for (Map.Entry<Long, Long> rule : initialconflictRules.entrySet()) {
+ if (LongBitmap.isOverlap(tes, rule.getKey())) {
+ tes |= rule.getValue();
+ continue;
+ }
+ conflictRulesAfterAbsorb.put(rule.getKey(), rule.getValue());
+ }
+ return tes;
+ }
+
+ /**
+ * Check whether the operator is applicable through the conflict rule.
+ *
+ * @param subGraph table set of the current join operator
Review Comment:
there is no "current join operator" in this context, only the caller has it
That's why this should also probably be a method of the hypergraph class.
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperGraph.java:
##########
@@ -401,69 +453,67 @@ public RexNode extractJoinCond(RelNode left, RelNode right, List<HyperEdge> edge
};
for (HyperEdge edge : edges) {
- RexNode inputRefCond = edge.getCondition().accept(inputName2InputRefShuttle);
+ RexNode inputRefCond = edge.getCondition().accept(shuttle);
joinConds.add(inputRefCond);
}
- return RexUtil.composeConjunction(left.getCluster().getRexBuilder(), joinConds);
+ return RexUtil.composeConjunction(getCluster().getRexBuilder(), joinConds);
}
/**
- * Before starting enumeration, add Project on every input, make all field name unique.
- * Convert the HyperEdge condition from RexInputRef to RexInputFieldName
+ * Restore the projection order of the final result to the original plan.
+ *
+ * @param resultOrder the node order of the final result
+ * @param rowTypeList rowType of the final result
+ * @return list of RexInputRef
*/
- public void convertHyperEdgeCond(RelBuilder builder) {
- int fieldIndex = 0;
- List<RelDataTypeField> fieldList = rowType.getFieldList();
- for (int nodeIndex = 0; nodeIndex < inputs.size(); nodeIndex++) {
- RelNode input = inputs.get(nodeIndex);
- List<RexNode> projects = new ArrayList<>();
- List<String> names = new ArrayList<>();
- for (int i = 0; i < input.getRowType().getFieldCount(); i++) {
- projects.add(
- new RexInputRef(
- i,
- fieldList.get(fieldIndex).getType()));
- names.add(fieldList.get(fieldIndex).getName());
- fieldIndex++;
+ public List<RexNode> restoreProjectionOrder(
+ ImmutableList<Integer> resultOrder,
+ List<RelDataTypeField> rowTypeList) {
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
+ int fieldCount = 0;
+ for (int resultIndex : resultOrder) {
+ relativePositionInNode.put(resultIndex, fieldCount);
+ if (!LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(resultIndex))) {
+ fieldCount += inputs.get(resultIndex).getRowType().getFieldCount();
}
-
- builder.push(input)
- .project(projects, names, true);
- replaceInput(nodeIndex, builder.build());
}
-
- RexShuttle inputRef2inputNameShuttle = new RexShuttle() {
- @Override public RexNode visitInputRef(RexInputRef inputRef) {
- int index = inputRef.getIndex();
- return new RexInputFieldName(
- fieldList.get(index).getName(),
- fieldList.get(index).getType());
+ List<RexNode> projects = new ArrayList<>();
+ for (int inputIndex = 0; inputIndex < inputs.size(); inputIndex++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(inputIndex))) {
+ continue;
}
- };
- for (int i = 0; i < edges.size(); i++) {
- HyperEdge edge = edges.get(i);
- RexNode convertCond = edge.getCondition().accept(inputRef2inputNameShuttle);
- HyperEdge convertEdge =
- new HyperEdge(
- edge.getLeftNodeBitmap(),
- edge.getRightNodeBitmap(),
- edge.getJoinType(),
- convertCond);
- edges.set(i, convertEdge);
+ for (int i = 0; i < inputs.get(inputIndex).getRowType().getFieldCount(); i++) {
+ Integer fieldOffset = relativePositionInNode.get(inputIndex);
+ if (fieldOffset == null) {
+ throw new DpHyp.DphypOrHyperGraphException(
+ "The result order loses the " + inputIndex + "-th input");
Review Comment:
looses?
is this an assertion failure?
Then you should probably use the AssertionFailure exception.
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperGraph.java:
##########
@@ -369,25 +388,58 @@ private static boolean isOverlapEdge(HyperEdge edge, long subset) {
return joinType;
}
- public RexNode extractJoinCond(RelNode left, RelNode right, List<HyperEdge> edges) {
+ /**
+ * Restore join condition from hyper edges.
Review Comment:
I think that a better description is "build an RexNode expression for the predicate corresponding to a set of hyperedges".
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * On the correct and complete enumeration of the core search space.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
Review Comment:
Instead of TODO I would write in the comment that the table here assumes that the predicate are assumed not to be null-rejecting.
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -37,7 +40,12 @@ public class DpHyp {
private final HyperGraph hyperGraph;
- private final HashMap<Long, RelNode> dpTable;
+ private final Map<Long, RelNode> dpTable;
+
+ // record the node order corresponding to the best subgraph, which is used to convert
+ // the RexNodeAndFieldIndex in hyperedge to the RexInputRef in join condition, and permute
+ // final result
+ private final Map<Long, ImmutableList<Integer>> resultInputOrder;
Review Comment:
this map can become very large, is that right?
is there any mechanism planned to control size explosion?
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperEdge.java:
##########
@@ -20,42 +20,96 @@
import org.apache.calcite.rel.core.JoinRelType;
import org.apache.calcite.rex.RexNode;
+import java.util.HashMap;
+import java.util.Map;
+
/**
* Edge in HyperGraph, that represents a join predicate.
*/
@Experimental
public class HyperEdge {
- private final long leftNodeBits;
+ // equivalent to the l-tes in CD-C paper
+ private final long leftEndpoint;
Review Comment:
if these are all bitmaps, you should use a suitable bitmap class. Calcite already has a BitSet class. On second thought, using a class may be too expensive.
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
+ List<HyperEdge> leftSubEdges,
+ List<HyperEdge> rightSubEdges,
+ JoinRelType joinType) {
+ Map<Long, Long> conflictRules = new HashMap<>();
Review Comment:
Can you define a class for a ConflictRule, even if it only stores a `HashMap<Long,Long>` inside?
The documentation and methods of this class will make it much clearer about what about the fact that these Longs are bitmaps. Also, these ConflictRules should point to the hypergraph, and have nice toString() methods which translate indexes in the bitmap into operators in the plan.
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
+ List<HyperEdge> leftSubEdges,
+ List<HyperEdge> rightSubEdges,
+ JoinRelType joinType) {
+ Map<Long, Long> conflictRules = new HashMap<>();
+ for (HyperEdge leftSubEdge : leftSubEdges) {
+ if (!isAssociative(leftSubEdge.getJoinType(), joinType)) {
+ if (leftSubEdge.getLeftNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ leftSubEdge.getInitialRightNodeBits(),
+ leftSubEdge.getLeftNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ leftSubEdge.getInitialRightNodeBits(),
+ leftSubEdge.getInitialLeftNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+
+ }
+ if (!isLeftAsscom(leftSubEdge.getJoinType(), joinType)) {
+ if (leftSubEdge.getRightNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ leftSubEdge.getInitialLeftNodeBits(),
+ leftSubEdge.getRightNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ leftSubEdge.getInitialLeftNodeBits(),
+ leftSubEdge.getInitialRightNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ }
+
+ for (HyperEdge rightSubEdge : rightSubEdges) {
+ if (!isAssociative(joinType, rightSubEdge.getJoinType())) {
+ if (rightSubEdge.getRightNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ rightSubEdge.getInitialLeftNodeBits(),
+ rightSubEdge.getRightNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ rightSubEdge.getInitialLeftNodeBits(),
+ rightSubEdge.getInitialRightNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ if (!isRightAsscom(joinType, rightSubEdge.getJoinType())) {
+ if (rightSubEdge.getLeftNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ rightSubEdge.getInitialRightNodeBits(),
+ rightSubEdge.getLeftNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ rightSubEdge.getInitialRightNodeBits(),
+ rightSubEdge.getInitialLeftNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ }
+ return conflictRules;
+ }
+
+ /**
+ * For conflict rule <code>T1 -> T2</code>, if T1 and tes have intersection, we can add T2
+ * into tes and remove this rule. See section 5.5 in paper.
+ *
+ * @param tes tes
Review Comment:
at least here you should spell Total Eligibility Set.
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperGraph.java:
##########
@@ -401,69 +453,67 @@ public RexNode extractJoinCond(RelNode left, RelNode right, List<HyperEdge> edge
};
for (HyperEdge edge : edges) {
- RexNode inputRefCond = edge.getCondition().accept(inputName2InputRefShuttle);
+ RexNode inputRefCond = edge.getCondition().accept(shuttle);
joinConds.add(inputRefCond);
}
- return RexUtil.composeConjunction(left.getCluster().getRexBuilder(), joinConds);
+ return RexUtil.composeConjunction(getCluster().getRexBuilder(), joinConds);
}
/**
- * Before starting enumeration, add Project on every input, make all field name unique.
- * Convert the HyperEdge condition from RexInputRef to RexInputFieldName
+ * Restore the projection order of the final result to the original plan.
+ *
+ * @param resultOrder the node order of the final result
+ * @param rowTypeList rowType of the final result
+ * @return list of RexInputRef
*/
- public void convertHyperEdgeCond(RelBuilder builder) {
- int fieldIndex = 0;
- List<RelDataTypeField> fieldList = rowType.getFieldList();
- for (int nodeIndex = 0; nodeIndex < inputs.size(); nodeIndex++) {
- RelNode input = inputs.get(nodeIndex);
- List<RexNode> projects = new ArrayList<>();
- List<String> names = new ArrayList<>();
- for (int i = 0; i < input.getRowType().getFieldCount(); i++) {
- projects.add(
- new RexInputRef(
- i,
- fieldList.get(fieldIndex).getType()));
- names.add(fieldList.get(fieldIndex).getName());
- fieldIndex++;
+ public List<RexNode> restoreProjectionOrder(
+ ImmutableList<Integer> resultOrder,
+ List<RelDataTypeField> rowTypeList) {
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
+ int fieldCount = 0;
+ for (int resultIndex : resultOrder) {
+ relativePositionInNode.put(resultIndex, fieldCount);
+ if (!LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(resultIndex))) {
+ fieldCount += inputs.get(resultIndex).getRowType().getFieldCount();
}
-
- builder.push(input)
- .project(projects, names, true);
- replaceInput(nodeIndex, builder.build());
}
-
- RexShuttle inputRef2inputNameShuttle = new RexShuttle() {
- @Override public RexNode visitInputRef(RexInputRef inputRef) {
- int index = inputRef.getIndex();
- return new RexInputFieldName(
- fieldList.get(index).getName(),
- fieldList.get(index).getType());
+ List<RexNode> projects = new ArrayList<>();
+ for (int inputIndex = 0; inputIndex < inputs.size(); inputIndex++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(inputIndex))) {
+ continue;
}
- };
- for (int i = 0; i < edges.size(); i++) {
- HyperEdge edge = edges.get(i);
- RexNode convertCond = edge.getCondition().accept(inputRef2inputNameShuttle);
- HyperEdge convertEdge =
- new HyperEdge(
- edge.getLeftNodeBitmap(),
- edge.getRightNodeBitmap(),
- edge.getJoinType(),
- convertCond);
- edges.set(i, convertEdge);
+ for (int i = 0; i < inputs.get(inputIndex).getRowType().getFieldCount(); i++) {
+ Integer fieldOffset = relativePositionInNode.get(inputIndex);
+ if (fieldOffset == null) {
+ throw new DpHyp.DphypOrHyperGraphException(
+ "The result order loses the " + inputIndex + "-th input");
+ }
+ int inputRef = i + fieldOffset;
+ projects.add(
+ new RexInputRef(inputRef, rowTypeList.get(inputRef).getType()));
+ }
}
+ return projects;
}
/**
* Adjusting RexInputRef in enumeration process is too complicated,
- * so use unique name replace input ref.
- * Before starting enumeration, convert RexInputRef to RexInputFieldName.
- * When connect csgcmp to Join, convert RexInputFieldName to RexInputRef.
+ * so use node index and relative position of field in node replace RexInputRef.
+ * When build hyper graph, convert RexInputRef to RexNodeAndFieldIndex.
+ * When connect csgcmp to Join, convert RexNodeAndFieldIndex to RexInputRef.
*/
- private static class RexInputFieldName extends RexVariable {
+ static class RexNodeAndFieldIndex extends RexVariable {
Review Comment:
why is this a static class if the nodeIndex depends on the hypergraph?
Should be instead an inner class of the HyperGraph?
Then you could access the RexNode too.
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -173,42 +180,86 @@ private void enumerateCmpRec(long csg, long cmp, long forbidden) {
private void emitCsgCmp(long csg, long cmp, List<HyperEdge> edges) {
RelNode child1 = dpTable.get(csg);
RelNode child2 = dpTable.get(cmp);
+ ImmutableList csgOrder = resultInputOrder.get(csg);
+ ImmutableList cmpOrder = resultInputOrder.get(cmp);
if (child1 == null || child2 == null) {
- throw new IllegalArgumentException(
+ throw new DphypOrHyperGraphException(
"csg and cmp were not enumerated in the previous dp process");
}
+ if (csgOrder == null || cmpOrder == null) {
+ throw new DphypOrHyperGraphException("Lost the vertex order of csg or cmp");
+ }
JoinRelType joinType = hyperGraph.extractJoinType(edges);
if (joinType == null) {
return;
}
- RexNode joinCond1 = hyperGraph.extractJoinCond(child1, child2, edges);
+ if (!ConflictDetectionHelper.applicable(csg | cmp, edges)) {
+ return;
+ }
+
+ ImmutableList<Integer> unionOrder = ImmutableList.<Integer>builder()
+ .addAll(csgOrder)
+ .addAll(cmpOrder)
+ .build();
+ RexNode joinCond1 = hyperGraph.extractJoinCond(unionOrder, csgOrder.size(), edges);
RelNode newPlan1 = builder
.push(child1)
.push(child2)
.join(joinType, joinCond1)
.build();
+ RelNode winPlan = newPlan1;
+ ImmutableList<Integer> winOrder = ImmutableList.copyOf(unionOrder);
- // swap left and right
- RexNode joinCond2 = hyperGraph.extractJoinCond(child2, child1, edges);
- RelNode newPlan2 = builder
- .push(child2)
- .push(child1)
- .join(joinType, joinCond2)
- .build();
- RelNode winPlan = chooseBetterPlan(newPlan1, newPlan2);
+ if (ConflictDetectionHelper.isCommutative(joinType)) {
+ // swap left and right
+ unionOrder = ImmutableList.<Integer>builder()
+ .addAll(cmpOrder)
+ .addAll(csgOrder)
+ .build();
+ RexNode joinCond2 = hyperGraph.extractJoinCond(unionOrder, cmpOrder.size(), edges);
+ RelNode newPlan2 = builder
+ .push(child2)
+ .push(child1)
+ .join(joinType, joinCond2)
+ .build();
+ winPlan = chooseBetterPlan(winPlan, newPlan2);
+ if (winPlan.equals(newPlan2)) {
+ winOrder = ImmutableList.copyOf(unionOrder);
+ }
+ }
RelNode oriPlan = dpTable.get(csg | cmp);
if (oriPlan != null) {
winPlan = chooseBetterPlan(winPlan, oriPlan);
+ if (winPlan.equals(oriPlan)) {
+ winOrder = resultInputOrder.get(csg | cmp);
+ }
}
+ assert winOrder != null;
dpTable.put(csg | cmp, winPlan);
+ resultInputOrder.put(csg | cmp, winOrder);
}
public @Nullable RelNode getBestPlan() {
int size = hyperGraph.getInputs().size();
long wholeGraph = LongBitmap.newBitmapBetween(0, size);
- return dpTable.get(wholeGraph);
+ RelNode orderedJoin = dpTable.get(wholeGraph);
+ if (orderedJoin == null) {
+ return null;
+ }
+ ImmutableList<Integer> resultOrder = resultInputOrder.get(wholeGraph);
+ if (resultOrder == null) {
Review Comment:
can this happen for legal programs, or this is essentially an assertion failure?
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,106 +75,205 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ rightEndpoint = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
- edges,
+ notProjectInputs,
+ leftSubEdges,
origJoin.getRowType());
call.transformTo(result);
}
- private static HyperEdge adjustNodeBit(HyperEdge hyperEdge, int nodeOffset, int fieldOffset) {
- RexNode newCondition = RexUtil.shift(hyperEdge.getCondition(), fieldOffset);
+ private static HyperEdge adjustNodeBit(HyperEdge hyperEdge, int nodeOffset) {
Review Comment:
should this be a method of the HyperEdge class?
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperGraph.java:
##########
@@ -401,69 +453,67 @@ public RexNode extractJoinCond(RelNode left, RelNode right, List<HyperEdge> edge
};
for (HyperEdge edge : edges) {
- RexNode inputRefCond = edge.getCondition().accept(inputName2InputRefShuttle);
+ RexNode inputRefCond = edge.getCondition().accept(shuttle);
joinConds.add(inputRefCond);
}
- return RexUtil.composeConjunction(left.getCluster().getRexBuilder(), joinConds);
+ return RexUtil.composeConjunction(getCluster().getRexBuilder(), joinConds);
Review Comment:
I should have looked at the original hypergraph PR, but I haven't.
It seems to me like the shape of predicates in all joins is restricted to be conjunctions of equalities involving simple field accesses. Is this right?
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperGraph.java:
##########
@@ -401,69 +453,67 @@ public RexNode extractJoinCond(RelNode left, RelNode right, List<HyperEdge> edge
};
for (HyperEdge edge : edges) {
- RexNode inputRefCond = edge.getCondition().accept(inputName2InputRefShuttle);
+ RexNode inputRefCond = edge.getCondition().accept(shuttle);
joinConds.add(inputRefCond);
}
- return RexUtil.composeConjunction(left.getCluster().getRexBuilder(), joinConds);
+ return RexUtil.composeConjunction(getCluster().getRexBuilder(), joinConds);
}
/**
- * Before starting enumeration, add Project on every input, make all field name unique.
- * Convert the HyperEdge condition from RexInputRef to RexInputFieldName
+ * Restore the projection order of the final result to the original plan.
Review Comment:
ensure that the fields produced by the reordered join are in the same order as in the original plan.
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,106 +75,205 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ rightEndpoint = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
- edges,
+ notProjectInputs,
+ leftSubEdges,
origJoin.getRowType());
call.transformTo(result);
}
- private static HyperEdge adjustNodeBit(HyperEdge hyperEdge, int nodeOffset, int fieldOffset) {
- RexNode newCondition = RexUtil.shift(hyperEdge.getCondition(), fieldOffset);
+ private static HyperEdge adjustNodeBit(HyperEdge hyperEdge, int nodeOffset) {
+ RexShuttle shiftNodeIndexShuttle = new RexShuttle() {
+ @Override protected List<RexNode> visitList(
+ List<? extends RexNode> exprs,
+ boolean @Nullable [] update) {
+ ImmutableList.Builder<RexNode> clonedOperands = ImmutableList.builder();
+ for (RexNode operand : exprs) {
+ RexNode clonedOperand;
+ if (operand instanceof HyperGraph.RexNodeAndFieldIndex) {
+ clonedOperand =
+ new HyperGraph.RexNodeAndFieldIndex(
+ ((HyperGraph.RexNodeAndFieldIndex) operand).nodeIndex + nodeOffset,
Review Comment:
you should cast the operand only once
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
+ List<HyperEdge> leftSubEdges,
+ List<HyperEdge> rightSubEdges,
+ JoinRelType joinType) {
+ Map<Long, Long> conflictRules = new HashMap<>();
+ for (HyperEdge leftSubEdge : leftSubEdges) {
+ if (!isAssociative(leftSubEdge.getJoinType(), joinType)) {
+ if (leftSubEdge.getLeftNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ leftSubEdge.getInitialRightNodeBits(),
+ leftSubEdge.getLeftNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ leftSubEdge.getInitialRightNodeBits(),
+ leftSubEdge.getInitialLeftNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+
+ }
+ if (!isLeftAsscom(leftSubEdge.getJoinType(), joinType)) {
+ if (leftSubEdge.getRightNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ leftSubEdge.getInitialLeftNodeBits(),
+ leftSubEdge.getRightNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ leftSubEdge.getInitialLeftNodeBits(),
+ leftSubEdge.getInitialRightNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ }
+
+ for (HyperEdge rightSubEdge : rightSubEdges) {
+ if (!isAssociative(joinType, rightSubEdge.getJoinType())) {
+ if (rightSubEdge.getRightNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ rightSubEdge.getInitialLeftNodeBits(),
+ rightSubEdge.getRightNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ rightSubEdge.getInitialLeftNodeBits(),
+ rightSubEdge.getInitialRightNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ if (!isRightAsscom(joinType, rightSubEdge.getJoinType())) {
+ if (rightSubEdge.getLeftNodeUsedInPredicate() != 0) {
+ conflictRules.merge(
+ rightSubEdge.getInitialRightNodeBits(),
+ rightSubEdge.getLeftNodeUsedInPredicate(),
+ (oldValue, newValue) -> oldValue | newValue);
+ } else {
+ conflictRules.merge(
+ rightSubEdge.getInitialRightNodeBits(),
+ rightSubEdge.getInitialLeftNodeBits(),
+ (oldValue, newValue) -> oldValue | newValue);
+ }
+ }
+ }
+ return conflictRules;
+ }
+
+ /**
+ * For conflict rule <code>T1 -> T2</code>, if T1 and tes have intersection, we can add T2
+ * into tes and remove this rule. See section 5.5 in paper.
+ *
+ * @param tes tes
Review Comment:
I would also make a class for TES, this will make it much easier to read the code.
This then becomes a method of the TES class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2113331471
##########
core/src/main/java/org/apache/calcite/rel/rules/HyperGraph.java:
##########
@@ -401,69 +453,67 @@ public RexNode extractJoinCond(RelNode left, RelNode right, List<HyperEdge> edge
};
for (HyperEdge edge : edges) {
- RexNode inputRefCond = edge.getCondition().accept(inputName2InputRefShuttle);
+ RexNode inputRefCond = edge.getCondition().accept(shuttle);
joinConds.add(inputRefCond);
}
- return RexUtil.composeConjunction(left.getCluster().getRexBuilder(), joinConds);
+ return RexUtil.composeConjunction(getCluster().getRexBuilder(), joinConds);
Review Comment:
Yes, they are okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2958708077
Great job! However, more test cases need to be added to make it convincing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2918563812
> No, I don't mean RelOptRulesTests. You need something that looks like unit tests for your code, not for Calcite. For example, when you enumerate candidate plans, it would be nice to validate that these plans are all valid and generate the same result, even if they are not optimal. (Checking that the result is correct may not be easy, though, given the current testing infrastructure.)
I completely agree with your point, including your last point, I think it is indeed difficult to complete it in the current testing infrastructure. I don't have a good idea. Maybe create a new test file specifically for testing the execution results of dphyp?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2113654418
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
Review Comment:
Got it. But before calling this method, `JoinToHyperGraphRule` has already checked the type of the left and right children of the Join node. If it is a `HyperGraph`, it extracts the hyperedges to leftSubEdges and rightSubEdges, and shifts the bitmap for rightSubEdges. If this method only takes Join node, then the above operations need to be repeated, so it takes three necessary arguments here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2115108148
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
Review Comment:
```suggestion
* <a href="https://dl.acm.org/doi/pdf/10.1145/2463676.2465314">
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2116144373
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,104 +75,160 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> hyperEdge.adjustNodeBit(leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new AssertionError("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new AssertionError("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new AssertionError("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = initialLeftNodeBits;
+ rightEndpoint = initialRightNodeBits;
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
Review Comment:
Thank you very much for pointing this out! This is indeed a bug in the current code. Inner join can be rewritten as cross product + where. Due to this property, inner join condition can be split by conjunction and freely combine with other inner join conditions. This privilege cannot be extended to non-inner joins, so we cannot split the condition of non-inner joins, that is, only generate one edge.
The following example may be more clear:
```
t1
id name
1 t1
2 t1
t2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2103865329
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * On the correct and complete enumeration of the core search space.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
Review Comment:
The associativity table given in the paper is shown in the figure below. Assuming that o_a and o_b are both left joins, null rejection occurs on the left child of o_b. I think this is not a case where outer join will be rewritten as inner join, so I recorded this TODO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2982970419
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392) **Quality Gate passed**
Issues
 [13 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [74.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2156030199
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -173,42 +202,140 @@ private void enumerateCmpRec(long csg, long cmp, long forbidden) {
private void emitCsgCmp(long csg, long cmp, List<HyperEdge> edges) {
RelNode child1 = dpTable.get(csg);
RelNode child2 = dpTable.get(cmp);
- if (child1 == null || child2 == null) {
- throw new IllegalArgumentException(
- "csg and cmp were not enumerated in the previous dp process");
- }
+ ImmutableList<HyperGraph.NodeState> csgOrder = resultInputOrder.get(csg);
+ ImmutableList<HyperGraph.NodeState> cmpOrder = resultInputOrder.get(cmp);
+ assert child1 != null && child2 != null && csgOrder != null && cmpOrder != null;
+ assert Long.bitCount(csg) == csgOrder.size() && Long.bitCount(cmp) == cmpOrder.size();
JoinRelType joinType = hyperGraph.extractJoinType(edges);
if (joinType == null) {
return;
}
- RexNode joinCond1 = hyperGraph.extractJoinCond(child1, child2, edges);
+ // verify whether the subgraph is legal by using the conflict rules in hyperedges
+ if (!hyperGraph.applicable(csg | cmp, edges)) {
+ return;
+ }
+
+ List<HyperGraph.NodeState> unionOrder = new ArrayList<>(csgOrder);
+ unionOrder.addAll(cmpOrder);
+ // build join condition from hyperedges. e.g.
+ // case.1
+ // csg: node0_projected [field0, field1], node1_projected [field0, field1],
+ //
+ // join
+ // / \
+ // node0 node1
+ //
+ // cmp: node2_projected [field0, field1]
+ // hyperedge1: node0.field0 = node2.field0
+ // hyperedge2: node1.field1 = node2.field1
+ // we will get join condition: ($0 = $4) and ($3 = $5)
+ //
+ // new_join(condition=[AND(($0 = $4), ($3 = $5))])
+ // / \
+ // join node2
+ // / \
+ // node0 node1
+ //
+ // case.2
+ // csg: node0_projected [field0, field1], node1_not_projected [field0, field1],
+ //
+ // join(joinType=semi/anti)
+ // / \
+ // node0 node1
+ //
+ // cmp: node2_projected [field0, field1]
+ // hyperedge1: node0.field0 = node2.field0
+ // hyperedge2: node0.field1 = node2.field1
+ // we will get join condition: ($0 = $2) and ($1 = $3)
+ //
+ // new_join(condition=[AND(($0 = $2), ($1 = $3))])
+ // / \
+ // join(joinType=semi/anti) node2
+ // / \
+ // node0 node1
+ RexNode joinCond1 = hyperGraph.extractJoinCond(unionOrder, csgOrder.size(), edges, joinType);
RelNode newPlan1 = builder
.push(child1)
.push(child2)
.join(joinType, joinCond1)
.build();
+ RelNode winPlan = newPlan1;
+ ImmutableList<HyperGraph.NodeState> winOrder = ImmutableList.copyOf(unionOrder);
+ assert verifyDpResultRowType(newPlan1, unionOrder);
Review Comment:
I see, I was looking for a test, but this is better.
But can you also write a test that returns all the plans for a query and executes them and compares results?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2155823161
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -55,119 +53,146 @@ protected JoinToHyperGraphRule(Config config) {
final Join origJoin = call.rel(0);
final RelNode left = call.rel(1);
final RelNode right = call.rel(2);
- if (origJoin.getJoinType() != JoinRelType.INNER) {
+ if (unSupportedJoinType(origJoin.getJoinType())) {
Review Comment:
You're right, I'll update it. :)
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -1643,6 +1670,33 @@ LogicalProject(EMPNO=[$9])
LogicalJoin(condition=[=($3, $0)], joinType=[inner])
LogicalTableScan(table=[[CATALOG, SALES, EMP_ADDRESS]])
LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+]]>
+ </Resource>
+ </TestCase>
+ <TestCase name="testComplexPredicateForDphyp">
+ <Resource name="sql">
+ <![CDATA[select emp.empno from emp, emp_b, dept, dept_nested where emp.deptno + emp_b.empno = dept.deptno + dept_nested.deptno and (emp.empno = emp_b.empno or emp.ename = emp_b.ename) and sqrt(dept.deptno + dept_nested.deptno) > 2]]>
+ </Resource>
+ <Resource name="planBefore">
+ <![CDATA[
+LogicalProject(EMPNO=[$0])
+ HyperGraph(edges=[{0, 1, 2}——[INNER, =(+(node(0)_field(7), node(1)_field(0)), +(node(2)_field(0), node(3)_field(0)))]——{3},{2}——[INNER, >(POWER(+(node(2)_field(0), node(3)_field(0)), 0.5:DECIMAL(2, 1)), CAST(2):DOUBLE NOT NULL)]——{3},{0, 1}——[INNER, true]——{2},{0}——[INNER, OR(=(node(0)_field(0), node(1)_field(0)), =(node(0)_field(1), node(1)_field(1)))]——{1}])
Review Comment:
The print of hyperedge is indeed a bit long. But I think it is necessary to print the endpoints, join type and condition of hyperedge, I haven't thought of a better form yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2163054401
##########
core/src/test/java/org/apache/calcite/test/DphypJoinReorderTest.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.test;
+
+import org.apache.calcite.DataContexts;
+import org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.calcite.adapter.enumerable.EnumerableInterpretable;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRules;
+import org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRules;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.hep.HepMatchOrder;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.plan.hep.HepProgramBuilder;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.rules.DpHyp;
+import org.apache.calcite.rel.rules.HyperGraph;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.runtime.Bindable;
+import org.apache.calcite.tools.RelBuilder;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+public class DphypJoinReorderTest {
+
+ @Test void testInnerLeftSemiJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // left join t3 on id2=id3
+ // where exists (select * from t4 where id3=id4)
+ Function<RelBuilder, RelNode> function = builder ->
Review Comment:
This is a great start to construct different join test cases in this way. Based on this, we can add more complex test cases to make the code more robust!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2165713703
##########
core/src/test/java/org/apache/calcite/test/DphypJoinReorderTest.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.test;
+
+import org.apache.calcite.DataContexts;
+import org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.calcite.adapter.enumerable.EnumerableInterpretable;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRules;
+import org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRules;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.hep.HepMatchOrder;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.plan.hep.HepProgramBuilder;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.rules.DpHyp;
+import org.apache.calcite.rel.rules.HyperGraph;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.runtime.Bindable;
+import org.apache.calcite.tools.RelBuilder;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+public class DphypJoinReorderTest {
+
+ @Test void testInnerLeftSemiJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // left join t3 on id2=id3
+ // where exists (select * from t4 where id3=id4)
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.LEFT,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 1, "Tom", 3, "Li")
+ .semiJoin(
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult = "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testChainInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id2=id3
+ // inner join t4 on id3=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, null, 3, "Li")
+ .values(new String[]{"id2", "name2"}, 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=2; name1=null; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testStarInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id1=id3
+ // inner join t4 on id1=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy", 1, "Tom")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom; id4=1; name4=Tom\n"
+ + "id1=2; name1=Lucy; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ /**
+ * Run the customized plan and check whether its execution results meet expectations (ignore row order).
+ *
+ * @param customPlanFunction a function that accepts one RelBuilder and produces a RelNode
+ * @param expectedResult expected result
+ */
+ void run(Function<RelBuilder, RelNode> customPlanFunction, String expectedResult) {
+ VolcanoPlanner volcanoPlanner = new VolcanoPlanner();
+ volcanoPlanner.addRelTraitDef(ConventionTraitDef.INSTANCE);
+ volcanoPlanner.addRelTraitDef(RelCollationTraitDef.INSTANCE);
+
+ // initialize plan
+ RelDataTypeFactory typeFactory =
+ new JavaTypeFactoryImpl();
+ RelOptCluster cluster = RelOptCluster.create(volcanoPlanner, new RexBuilder(typeFactory));
+ RelBuilder builder = RelFactories.LOGICAL_BUILDER.create(cluster, null);
+ RelNode initRel = customPlanFunction.apply(builder);
+
+ // convert to HyperGraph
+ HepProgram program = new HepProgramBuilder()
+ .addMatchOrder(HepMatchOrder.BOTTOM_UP)
+ .addRuleInstance(CoreRules.JOIN_TO_HYPER_GRAPH)
+ .build();
+ HepPlanner hepPlanner = new HepPlanner(program);
+ hepPlanner.setRoot(initRel);
+ RelNode hyperGraph = hepPlanner.findBestExp();
+ assert hyperGraph instanceof HyperGraph : "The root node must be a HyperGraph, "
+ + "please check the custom plan function. The root node now is:\n"
+ + RelOptUtil.toString(hyperGraph);
+
+ // start dphyp enumeration
+ DphypForTest dphyp =
+ new DphypForTest(
+ (HyperGraph) hyperGraph,
+ builder,
+ hyperGraph.getCluster().getMetadataQuery(),
+ 127);
+ dphyp.startEnumerateJoin();
+
+ // verify the execution results of each candidate plan enumerated by dphyp
+ for (RelNode candidatePlan : dphyp.candiateLists) {
+ for (RelOptRule enumerableRule : EnumerableRules.rules()) {
+ volcanoPlanner.addRule(enumerableRule);
+ }
+ for (RelOptRule calcRule : RelOptRules.CALC_RULES) {
+ volcanoPlanner.addRule(calcRule);
+ }
+ volcanoPlanner.removeRule(EnumerableRules.ENUMERABLE_PROJECT_RULE);
Review Comment:
please add this as a comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2111457179
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
+ List<HyperEdge> leftSubEdges,
+ List<HyperEdge> rightSubEdges,
+ JoinRelType joinType) {
+ Map<Long, Long> conflictRules = new HashMap<>();
Review Comment:
Defining the conflict rule as a class, as you said, it only stores a `HashMap<Long,Long>`, and I can't think of any other methods that need to be defined in this class (except toString). Is it worth? The conflict rules map is a field in `HyperEdge`, if we want to explain it more clearly, can we improve the comments in `HyperEdge`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2111463951
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -173,42 +180,86 @@ private void enumerateCmpRec(long csg, long cmp, long forbidden) {
private void emitCsgCmp(long csg, long cmp, List<HyperEdge> edges) {
RelNode child1 = dpTable.get(csg);
RelNode child2 = dpTable.get(cmp);
+ ImmutableList csgOrder = resultInputOrder.get(csg);
+ ImmutableList cmpOrder = resultInputOrder.get(cmp);
if (child1 == null || child2 == null) {
- throw new IllegalArgumentException(
+ throw new DphypOrHyperGraphException(
"csg and cmp were not enumerated in the previous dp process");
}
+ if (csgOrder == null || cmpOrder == null) {
+ throw new DphypOrHyperGraphException("Lost the vertex order of csg or cmp");
+ }
JoinRelType joinType = hyperGraph.extractJoinType(edges);
if (joinType == null) {
return;
}
- RexNode joinCond1 = hyperGraph.extractJoinCond(child1, child2, edges);
+ if (!ConflictDetectionHelper.applicable(csg | cmp, edges)) {
+ return;
+ }
+
+ ImmutableList<Integer> unionOrder = ImmutableList.<Integer>builder()
+ .addAll(csgOrder)
+ .addAll(cmpOrder)
+ .build();
+ RexNode joinCond1 = hyperGraph.extractJoinCond(unionOrder, csgOrder.size(), edges);
RelNode newPlan1 = builder
.push(child1)
.push(child2)
.join(joinType, joinCond1)
.build();
+ RelNode winPlan = newPlan1;
+ ImmutableList<Integer> winOrder = ImmutableList.copyOf(unionOrder);
- // swap left and right
- RexNode joinCond2 = hyperGraph.extractJoinCond(child2, child1, edges);
- RelNode newPlan2 = builder
- .push(child2)
- .push(child1)
- .join(joinType, joinCond2)
- .build();
- RelNode winPlan = chooseBetterPlan(newPlan1, newPlan2);
+ if (ConflictDetectionHelper.isCommutative(joinType)) {
+ // swap left and right
+ unionOrder = ImmutableList.<Integer>builder()
+ .addAll(cmpOrder)
+ .addAll(csgOrder)
+ .build();
+ RexNode joinCond2 = hyperGraph.extractJoinCond(unionOrder, cmpOrder.size(), edges);
+ RelNode newPlan2 = builder
+ .push(child2)
+ .push(child1)
+ .join(joinType, joinCond2)
+ .build();
+ winPlan = chooseBetterPlan(winPlan, newPlan2);
+ if (winPlan.equals(newPlan2)) {
+ winOrder = ImmutableList.copyOf(unionOrder);
+ }
+ }
RelNode oriPlan = dpTable.get(csg | cmp);
if (oriPlan != null) {
winPlan = chooseBetterPlan(winPlan, oriPlan);
+ if (winPlan.equals(oriPlan)) {
+ winOrder = resultInputOrder.get(csg | cmp);
+ }
}
+ assert winOrder != null;
dpTable.put(csg | cmp, winPlan);
+ resultInputOrder.put(csg | cmp, winOrder);
}
public @Nullable RelNode getBestPlan() {
int size = hyperGraph.getInputs().size();
long wholeGraph = LongBitmap.newBitmapBetween(0, size);
- return dpTable.get(wholeGraph);
+ RelNode orderedJoin = dpTable.get(wholeGraph);
+ if (orderedJoin == null) {
+ return null;
+ }
+ ImmutableList<Integer> resultOrder = resultInputOrder.get(wholeGraph);
+ if (resultOrder == null) {
Review Comment:
You are right. I have thrown exceptions in several places in the DpHyp related code, but they should not happen in a legal program. I think I should change them all to assertion failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2112399382
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
+ List<HyperEdge> leftSubEdges,
+ List<HyperEdge> rightSubEdges,
+ JoinRelType joinType) {
+ Map<Long, Long> conflictRules = new HashMap<>();
Review Comment:
My experience is that abstractions help code maintenance in the long run.
Everyone who needs to read this code will wonder what these long values are, but if they are in a class with a good name, that also documents the fields, it will be clear. This will also distinguish Maps<Long, Long> that represent conflict maps from other Maps with the same type signature that appear in the code.
And if you need more methods, you will discover that there is a place to add them, instead of having lots of static methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2113327152
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
+ List<HyperEdge> leftSubEdges,
+ List<HyperEdge> rightSubEdges,
+ JoinRelType joinType) {
+ Map<Long, Long> conflictRules = new HashMap<>();
Review Comment:
I will update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2965820711
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392) **Quality Gate passed**
Issues
 [13 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [74.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2163054401
##########
core/src/test/java/org/apache/calcite/test/DphypJoinReorderTest.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.test;
+
+import org.apache.calcite.DataContexts;
+import org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.calcite.adapter.enumerable.EnumerableInterpretable;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRules;
+import org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRules;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.hep.HepMatchOrder;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.plan.hep.HepProgramBuilder;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.rules.DpHyp;
+import org.apache.calcite.rel.rules.HyperGraph;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.runtime.Bindable;
+import org.apache.calcite.tools.RelBuilder;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+public class DphypJoinReorderTest {
+
+ @Test void testInnerLeftSemiJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // left join t3 on id2=id3
+ // where exists (select * from t4 where id3=id4)
+ Function<RelBuilder, RelNode> function = builder ->
Review Comment:
This is a great start to construct different join test cases in this way. Based on this, we can add more complex test cases to make the code more robust! And it's great to see that you've incorporated a small part of test cases from other databases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2893792417
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [62.6% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2893679101
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [62.6% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2098024944
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -1526,6 +1526,33 @@ LogicalJoin(condition=[=($1, $12)], joinType=[semi])
LogicalTableScan(table=[[scott, DEPT]])
LogicalTableScan(table=[[scott, EMP]])
LogicalTableScan(table=[[scott, BONUS]])
+]]>
+ </Resource>
+ </TestCase>
+ <TestCase name="testCartesianProductForDphyp">
+ <Resource name="sql">
+ <![CDATA[select emp.empno from emp_address, emp, emp_b, dept where emp_address.empno = emp.empno and dept.deptno = emp.deptno]]>
+ </Resource>
+ <Resource name="planBefore">
+ <![CDATA[
+LogicalProject(EMPNO=[$3])
+ HyperGraph(edges=[{0}——[INNER, =(vertex(0)_field(0), vertex(1)_field(0))]——{1},{0, 1}——[INNER, true]——{2},{1}——[INNER, =(vertex(3)_field(0), vertex(1)_field(7))]——{3}])
Review Comment:
Maybe in `HyperGraph` is better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2103865329
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * On the correct and complete enumeration of the core search space.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
Review Comment:
The associativity table given in the paper is shown in the figure below. Assuming that o_a and o_b are both left joins, null rejection occurs on the left child of o_b. I think this is not a case where outer join will be rewritten as inner join, so I recorded this TODO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2151884030
##########
core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java:
##########
@@ -10785,4 +10785,61 @@ private void checkLoptOptimizeJoinRule(LoptOptimizeJoinRule rule) {
.withRule(CoreRules.HYPER_GRAPH_OPTIMIZE, CoreRules.PROJECT_REMOVE, CoreRules.PROJECT_MERGE)
.check();
}
+
+ @Test void testVariousJoinTypeForDphyp() {
+ HepProgram program = new HepProgramBuilder()
+ .addMatchOrder(HepMatchOrder.BOTTOM_UP)
+ .addRuleInstance(CoreRules.PROJECT_MERGE)
+ .addRuleInstance(CoreRules.PROJECT_TO_SEMI_JOIN)
+ .addRuleInstance(CoreRules.JOIN_TO_HYPER_GRAPH)
+ .build();
+
+ sql("select emp.empno from "
+ + "emp_address inner join emp on emp_address.empno = emp.empno "
+ + "left join dept on emp.deptno = dept.deptno "
+ + "where exists(select * from dept_nested where dept.deptno = dept_nested.deptno)")
+ .withExpand(true)
+ .withDecorrelate(true)
+ .withPre(program)
+ .withRule(CoreRules.HYPER_GRAPH_OPTIMIZE, CoreRules.PROJECT_REMOVE, CoreRules.PROJECT_MERGE)
+ .check();
+ }
+
+ @Test void testVariousJoinTypeForDphyp2() {
Review Comment:
Do not use auto-increment numbers at the end of the test method name, you need to have a meaningful name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-3003311294
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392) **Quality Gate passed**
Issues
 [13 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [74.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu merged PR #4392:
URL: https://github.com/apache/calcite/pull/4392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2098009043
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -17826,16 +17911,16 @@ LogicalProject(EMPNO=[$0])
</Resource>
<Resource name="planAfter">
<![CDATA[
-LogicalProject(EMPNO=[$19])
- LogicalJoin(condition=[AND(=($19, $0), =(+($8, $0), +($17, $13)))], joinType=[inner])
- LogicalTableScan(table=[[CATALOG, SALES, EMP_ADDRESS]])
+LogicalProject(EMPNO=[$16])
Review Comment:
I think this is caused by double precision.
Before explaining this problem, I need to explain that in the previous pr, in order to conveniently represent `RexInputRef` during enumeration, a unique alias will be generated for each input before enumeration (that is, an additional Project operator will be generated).
However, relying on strings and input rowType will cause trouble when processing semi/anti joins (the right child will not be projected). Therefore, in this PR, I changed it to use node index and the relative position of the field in the node to represent `RexInputRef`.
So, even if the join order is essentially the same, the cost value of the plan in this pr and the previous pr is different during enumeration (the previous pr has an additional Project).
Specifically,
In the previous pr:
t1 join t2 cost=154.21564(with additional Project)
t2 join t1 cost=154.21564(with additional Project) winner
In this pr:
t1 join t2 cost=92.21564 winner
t2 join t1 cost=92.21564000000001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2098022296
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
Review Comment:
I will update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2098019159
##########
core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml:
##########
@@ -1526,6 +1526,33 @@ LogicalJoin(condition=[=($1, $12)], joinType=[semi])
LogicalTableScan(table=[[scott, DEPT]])
LogicalTableScan(table=[[scott, EMP]])
LogicalTableScan(table=[[scott, BONUS]])
+]]>
+ </Resource>
+ </TestCase>
+ <TestCase name="testCartesianProductForDphyp">
+ <Resource name="sql">
+ <![CDATA[select emp.empno from emp_address, emp, emp_b, dept where emp_address.empno = emp.empno and dept.deptno = emp.deptno]]>
+ </Resource>
+ <Resource name="planBefore">
+ <![CDATA[
+LogicalProject(EMPNO=[$3])
+ HyperGraph(edges=[{0}——[INNER, =(vertex(0)_field(0), vertex(1)_field(0))]——{1},{0, 1}——[INNER, true]——{2},{1}——[INNER, =(vertex(3)_field(0), vertex(1)_field(7))]——{3}])
Review Comment:
Sure, I will add javadoc in RelOptRulesTest.java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2103408396
##########
core/src/main/java/org/apache/calcite/rel/rules/JoinToHyperGraphRule.java:
##########
@@ -70,106 +75,205 @@ protected JoinToHyperGraphRule(Config config) {
RelOptUtil.decomposeConjunction(origJoin.getCondition(), joinConds);
}
- // when right is HyperGraph, need shift the leftNodeBit, rightNodeBit, condition of HyperEdge
+ // when right is HyperGraph, need shift fields related to bitmap of HyperEdge
int leftNodeCount;
- int leftFieldCount = left.getRowType().getFieldCount();
if (left instanceof HyperGraph && right instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.addAll(right.getInputs());
- edges.addAll(((HyperGraph) left).getEdges());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount))
.collect(Collectors.toList()));
} else if (left instanceof HyperGraph) {
leftNodeCount = left.getInputs().size();
inputs.addAll(left.getInputs());
inputs.add(right);
- edges.addAll(((HyperGraph) left).getEdges());
+ notProjectInputs |= ((HyperGraph) left).getNotProjectInputs();
+
+ leftSubEdges.addAll(((HyperGraph) left).getEdges());
} else if (right instanceof HyperGraph) {
leftNodeCount = 1;
inputs.add(left);
inputs.addAll(right.getInputs());
- edges.addAll(
+ notProjectInputs |= ((HyperGraph) right).getNotProjectInputs() << leftNodeCount;
+
+ rightSubEdges.addAll(
((HyperGraph) right).getEdges().stream()
- .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount, leftFieldCount))
+ .map(hyperEdge -> adjustNodeBit(hyperEdge, leftNodeCount))
.collect(Collectors.toList()));
} else {
leftNodeCount = 1;
inputs.add(left);
inputs.add(right);
}
- HashMap<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // calculate conflict rules
+ Map<Long, Long> conflictRules =
+ ConflictDetectionHelper.makeConflictRules(
+ leftSubEdges,
+ rightSubEdges,
+ origJoin.getJoinType());
+ leftSubEdges.addAll(rightSubEdges);
+
+ Map<Integer, Integer> fieldIndexToNodeIndexMap = new HashMap<>();
+ // the map from input index to the count of fields before it, used to convert RexInputRef to
+ // RexNodeAndFieldIndex
+ Map<Integer, Integer> relativePositionInNode = new HashMap<>();
int fieldCount = 0;
for (int i = 0; i < inputs.size(); i++) {
+ if (LongBitmap.isOverlap(notProjectInputs, LongBitmap.newBitmap(i))) {
+ continue;
+ }
+ relativePositionInNode.put(i, fieldCount);
for (int j = 0; j < inputs.get(i).getRowType().getFieldCount(); j++) {
fieldIndexToNodeIndexMap.put(fieldCount++, i);
}
}
// convert current join condition to hyper edge condition
for (RexNode joinCond : joinConds) {
- long leftNodeBits;
- long rightNodeBits;
+ long leftEndpoint;
+ long rightEndpoint;
+ long leftNodeUsedInPredicate;
+ long rightNodeUsedInPredicate;
+ long initialLeftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ long initialRightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
List<Integer> leftRefs = new ArrayList<>();
List<Integer> rightRefs = new ArrayList<>();
- RexVisitorImpl visitor = new RexVisitorImpl<Void>(true) {
- @Override public Void visitInputRef(RexInputRef inputRef) {
+ RexShuttle shuttle = new RexShuttle() {
+ @Override public RexNode visitInputRef(RexInputRef inputRef) {
Integer nodeIndex = fieldIndexToNodeIndexMap.get(inputRef.getIndex());
if (nodeIndex == null) {
- throw new IllegalArgumentException("RexInputRef refers a dummy field: "
- + inputRef + ", rowType is: " + origJoin.getRowType());
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, RexInputRef refers "
+ + "a dummy field: " + inputRef + ", rowType is: " + origJoin.getRowType());
}
if (nodeIndex < leftNodeCount) {
leftRefs.add(nodeIndex);
} else {
rightRefs.add(nodeIndex);
}
- return null;
+ Integer fieldOffset = relativePositionInNode.get(nodeIndex);
+ if (fieldOffset == null) {
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, failed to map "
+ + "input index to field count before it");
+ }
+ int fieldIndex = inputRef.getIndex() - fieldOffset;
+ if (fieldIndex < 0) {
+ throw new DpHyp.DphypOrHyperGraphException("When build hyper graph, failed to convert "
+ + "the input ref to the relative position of the field in the input");
+ }
+ return new HyperGraph.RexNodeAndFieldIndex(
+ nodeIndex,
+ fieldIndex,
+ inputRef.getName(),
+ inputRef.getType());
}
};
- joinCond.accept(visitor);
+ RexNode hyperEdgeCondition = joinCond.accept(shuttle);
- // when cartesian product, make it to complex hyper edge
+ Map<Long, Long> conflictRulesAfterAbsorb = new HashMap<>();
+ leftNodeUsedInPredicate = LongBitmap.newBitmapFromList(leftRefs);
+ rightNodeUsedInPredicate = LongBitmap.newBitmapFromList(rightRefs);
if (leftRefs.isEmpty() || rightRefs.isEmpty()) {
- leftNodeBits = LongBitmap.newBitmapBetween(0, leftNodeCount);
- rightNodeBits = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
+ // when cartesian product or degenerate predicate, a complex hyperedge is generated to fix
+ // current join operator without exploring more possibilities. See section 6.2 in CD-C paper
+ leftEndpoint = LongBitmap.newBitmapBetween(0, leftNodeCount);
+ rightEndpoint = LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
} else {
- leftNodeBits = LongBitmap.newBitmapFromList(leftRefs);
- rightNodeBits = LongBitmap.newBitmapFromList(rightRefs);
+ // simplify conflict rules. See section 5.5 in CD-C paper
+ long tes =
+ ConflictDetectionHelper.absorbConflictRulesIntoTES(
+ leftNodeUsedInPredicate | rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ conflictRules);
+ rightEndpoint = tes & initialRightNodeBits;
+ leftEndpoint = tes & ~rightEndpoint;
}
- edges.add(
+ leftSubEdges.add(
new HyperEdge(
- leftNodeBits,
- rightNodeBits,
+ leftEndpoint,
+ rightEndpoint,
+ leftNodeUsedInPredicate,
+ rightNodeUsedInPredicate,
+ conflictRulesAfterAbsorb,
+ initialLeftNodeBits,
+ initialRightNodeBits,
origJoin.getJoinType(),
- joinCond));
+ hyperEdgeCondition));
+ }
+
+ if (!origJoin.getJoinType().projectsRight()) {
+ notProjectInputs |= LongBitmap.newBitmapBetween(leftNodeCount, inputs.size());
}
result =
new HyperGraph(
origJoin.getCluster(),
origJoin.getTraitSet(),
inputs,
- edges,
+ notProjectInputs,
+ leftSubEdges,
origJoin.getRowType());
call.transformTo(result);
}
- private static HyperEdge adjustNodeBit(HyperEdge hyperEdge, int nodeOffset, int fieldOffset) {
- RexNode newCondition = RexUtil.shift(hyperEdge.getCondition(), fieldOffset);
+ private static HyperEdge adjustNodeBit(HyperEdge hyperEdge, int nodeOffset) {
+ RexShuttle shiftNodeIndexShuttle = new RexShuttle() {
+ @Override protected List<RexNode> visitList(
+ List<? extends RexNode> exprs,
+ boolean @Nullable [] update) {
+ ImmutableList.Builder<RexNode> clonedOperands = ImmutableList.builder();
+ for (RexNode operand : exprs) {
+ RexNode clonedOperand;
+ if (operand instanceof HyperGraph.RexNodeAndFieldIndex) {
+ clonedOperand =
+ new HyperGraph.RexNodeAndFieldIndex(
+ ((HyperGraph.RexNodeAndFieldIndex) operand).nodeIndex + nodeOffset,
+ ((HyperGraph.RexNodeAndFieldIndex) operand).fieldIndex,
+ ((HyperGraph.RexNodeAndFieldIndex) operand).getName(),
+ operand.getType());
+ } else {
+ clonedOperand = operand.accept(this);
+ }
+ if ((clonedOperand != operand) && (update != null)) {
+ update[0] = true;
+ }
+ clonedOperands.add(clonedOperand);
+ }
+ return clonedOperands.build();
+ }
+ };
+ RexNode newCondition = hyperEdge.getCondition().accept(shiftNodeIndexShuttle);
+ Map<Long, Long> newConflictRules = new HashMap<>();
+ for (Map.Entry<Long, Long> entry : hyperEdge.getConflictRules().entrySet()) {
+ newConflictRules.put(entry.getKey() << nodeOffset, entry.getValue() << nodeOffset);
+ }
return new HyperEdge(
- hyperEdge.getLeftNodeBitmap() << nodeOffset,
- hyperEdge.getRightNodeBitmap() << nodeOffset,
+ hyperEdge.getLeftEndpoint() << nodeOffset,
+ hyperEdge.getRightEndpoint() << nodeOffset,
+ hyperEdge.getLeftNodeUsedInPredicate() << nodeOffset,
+ hyperEdge.getRightNodeUsedInPredicate() << nodeOffset,
+ newConflictRules,
+ hyperEdge.getInitialLeftNodeBits() << nodeOffset,
+ hyperEdge.getInitialRightNodeBits() << nodeOffset,
hyperEdge.getJoinType(),
newCondition);
}
+ private static boolean unSupportedJoinType(JoinRelType joinType) {
+ return joinType == JoinRelType.RIGHT
Review Comment:
I find this surprising, I think RIGHT is symmetric with LEFT, and can be converted into a LEFT join by swapping inputs.
Is this the reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2101538165
##########
core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java:
##########
@@ -10720,4 +10720,52 @@ private void checkLoptOptimizeJoinRule(LoptOptimizeJoinRule rule) {
sql(sql).withRule(CoreRules.AGGREGATE_MIN_MAX_TO_LIMIT)
.check();
}
+
+ @Test void testOuterJoinForDphyp() {
Review Comment:
I am sorry to put forward my own opinion. I admire your algorithm transplantation ability and this algorithm may be very good, but referring to the standards of other computing engines, we may also need to introduce benchmarks or stress testing instances to determine how much performance improvement is obtained in what scenarios，the test cases do not include relevant benchmarks, so strictly speaking, it may not be possible to confirm that performance can be improved, so as to explain the benefits of this modification. @silundong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2112392792
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
Review Comment:
I assumed that from the join node you can get the left, right hyperedges and the join type, converting 3 arguments into 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2111849097
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
+ List<HyperEdge> leftSubEdges,
+ List<HyperEdge> rightSubEdges,
+ JoinRelType joinType) {
+ Map<Long, Long> conflictRules = new HashMap<>();
Review Comment:
I agree mihai. It may be possible to write it as an inner class to make the scope of the code clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2103879079
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * On the correct and complete enumeration of the core search space.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
Review Comment:
When o^a and o^b are both left join, for `e1 o^a_12 e2 o^b_23 e3`, they are associative only if the predicate of o^b is null-rejecting on the attributes of e2 (see Table 2 in the paper). I think this is not a case where outer join will be rewritten as inner join, so I recorded this TODO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2103865329
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * On the correct and complete enumeration of the core search space.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
Review Comment:
The associativity table given in the paper is shown in the figure below. Assuming that o_a and o_b are both left joins, null rejection occurs on the left child of o_b. I think this is not a case where outer join will be rewritten as inner join, so I recorded this TODO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2113358733
##########
core/src/main/java/org/apache/calcite/rel/rules/DpHyp.java:
##########
@@ -37,7 +40,12 @@ public class DpHyp {
private final HyperGraph hyperGraph;
- private final HashMap<Long, RelNode> dpTable;
+ private final Map<Long, RelNode> dpTable;
+
+ // record the node order corresponding to the best subgraph, which is used to convert
+ // the RexNodeAndFieldIndex in hyperedge to the RexInputRef in join condition, and permute
+ // final result
+ private final Map<Long, ImmutableList<Integer>> resultInputOrder;
Review Comment:
I don't think so. For example, if the graph is fully connected and any combination is legal, the size of the two map is 2^3-1=7 (the bitmap of 012 and 210 are both 7). For n vertices, I think the maximum number is 2^n-1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2114503487
##########
core/src/main/java/org/apache/calcite/rel/rules/ConflictDetectionHelper.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.rel.core.JoinRelType;
+
+import com.google.common.collect.ImmutableMap;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Conflict detection algorithm based on CD-C. More details are in paper:
+ * <a href="https://15721.courses.cs.cmu.edu/spring2019/papers/23-optimizer2/p493-moerkotte.pdf">
+ * On the correct and complete enumeration of the core search space</a>.
+ */
+public class ConflictDetectionHelper {
+
+ private ConflictDetectionHelper() {
+ }
+
+ private static final ImmutableMap<JoinRelType, Integer> INDEX_OF_TABLE =
+ ImmutableMap.<JoinRelType, Integer>builder()
+ .put(JoinRelType.INNER, 0).put(JoinRelType.SEMI, 1).put(JoinRelType.ANTI, 2)
+ .put(JoinRelType.LEFT, 3).put(JoinRelType.FULL, 4).build();
+
+ // TODO: when special attribute is null rejecting, left/full join is
+ // associative/left_asscom/right_asscom. See table2/3 in paper
+ private static final boolean[][] ASSOCIATIVE_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] LEFT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, true, true, true, false},
+ /* semi-A */ {true, true, true, true, false},
+ /* anti-A */ {true, true, true, true, false},
+ /* left-A */ {true, true, true, true, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ private static final boolean[][] RIGHT_ASSCOM_TABLE = {
+ // inner-B semi-B anti-B left-B full-B
+ /* inner-A */ {true, false, false, false, false},
+ /* semi-A */ {false, false, false, false, false},
+ /* anti-A */ {false, false, false, false, false},
+ /* left-A */ {false, false, false, false, false},
+ /* full-A */ {false, false, false, false, false}};
+
+ /**
+ * Make conflict rules for join operator based on CD-C.
+ *
+ * @param leftSubEdges left sub operators
+ * @param rightSubEdges right sub operators
+ * @param joinType current join operator
+ * @return a map from table set1 to table set2 that if set1 and the tables in the current join
+ * operator have an intersection, then set2 must be included in the current join operator
+ */
+ public static Map<Long, Long> makeConflictRules(
Review Comment:
if a change doesn't improve the readability or correctness, it doesn't need to be made.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2957868060
@mihaibudiu @xiedeyantu @iwanttobepowerful Thank you very much for your previous review and suggestions. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-2979722208
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392) **Quality Gate passed**
Issues
 [13 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [74.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-3001424575
@xiedeyantu if you think it's ready you can remove the label you added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4392:
URL: https://github.com/apache/calcite/pull/4392#issuecomment-3003467180
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392) **Quality Gate passed**
Issues
 [13 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4392&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4392&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [74.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4392&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4392)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2163054401
##########
core/src/test/java/org/apache/calcite/test/DphypJoinReorderTest.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.test;
+
+import org.apache.calcite.DataContexts;
+import org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.calcite.adapter.enumerable.EnumerableInterpretable;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRules;
+import org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRules;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.hep.HepMatchOrder;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.plan.hep.HepProgramBuilder;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.rules.DpHyp;
+import org.apache.calcite.rel.rules.HyperGraph;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.runtime.Bindable;
+import org.apache.calcite.tools.RelBuilder;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+public class DphypJoinReorderTest {
+
+ @Test void testInnerLeftSemiJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // left join t3 on id2=id3
+ // where exists (select * from t4 where id3=id4)
+ Function<RelBuilder, RelNode> function = builder ->
Review Comment:
This is a great start to construct different join test cases in this way. Based on this, we can add more complex test cases to make the code more robust! And it's great to see that you've incorporated test cases from other databases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-7029] Support DPhyp to handle various join types [calcite]
Posted by "silundong (via GitHub)" <gi...@apache.org>.
silundong commented on code in PR #4392:
URL: https://github.com/apache/calcite/pull/4392#discussion_r2165693416
##########
core/src/test/java/org/apache/calcite/test/DphypJoinReorderTest.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.test;
+
+import org.apache.calcite.DataContexts;
+import org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.calcite.adapter.enumerable.EnumerableInterpretable;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRules;
+import org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRules;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.hep.HepMatchOrder;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.plan.hep.HepProgramBuilder;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.rules.CoreRules;
+import org.apache.calcite.rel.rules.DpHyp;
+import org.apache.calcite.rel.rules.HyperGraph;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.runtime.Bindable;
+import org.apache.calcite.tools.RelBuilder;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+public class DphypJoinReorderTest {
+
+ @Test void testInnerLeftSemiJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // left join t3 on id2=id3
+ // where exists (select * from t4 where id3=id4)
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.LEFT,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 1, "Tom", 3, "Li")
+ .semiJoin(
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult = "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testChainInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id2=id3
+ // inner join t4 on id3=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, null, 3, "Li")
+ .values(new String[]{"id2", "name2"}, 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id2"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id3"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=2; name1=null; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ @Test void testStarInnerJoinReorder() {
+ // select * from t1 inner join t2 on id1=id2
+ // inner join t3 on id1=id3
+ // inner join t4 on id1=id4
+ Function<RelBuilder, RelNode> function = builder ->
+ builder
+ .values(new String[]{"id1", "name1"}, 1, "Tom", 2, "Lucy", 3, "Li")
+ .values(new String[]{"id2", "name2"}, 1, "Tom", 2, "Lucy")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id2")))
+ .values(new String[]{"id3", "name3"}, 1, "Tom", 2, "Lucy", 3, "Li", 4, "Peter")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id3")))
+ .values(new String[]{"id4", "name4"}, 2, "Lucy", 5, "Andy", 1, "Tom")
+ .join(
+ JoinRelType.INNER,
+ builder.equals(
+ builder.field(2, 0, "id1"),
+ builder.field(2, 1, "id4")))
+ .build();
+
+ String expectedResult =
+ "id1=1; name1=Tom; id2=1; name2=Tom; id3=1; name3=Tom; id4=1; name4=Tom\n"
+ + "id1=2; name1=Lucy; id2=2; name2=Lucy; id3=2; name3=Lucy; id4=2; name4=Lucy\n";
+ run(function, expectedResult);
+ }
+
+ /**
+ * Run the customized plan and check whether its execution results meet expectations (ignore row order).
+ *
+ * @param customPlanFunction a function that accepts one RelBuilder and produces a RelNode
+ * @param expectedResult expected result
+ */
+ void run(Function<RelBuilder, RelNode> customPlanFunction, String expectedResult) {
+ VolcanoPlanner volcanoPlanner = new VolcanoPlanner();
+ volcanoPlanner.addRelTraitDef(ConventionTraitDef.INSTANCE);
+ volcanoPlanner.addRelTraitDef(RelCollationTraitDef.INSTANCE);
+
+ // initialize plan
+ RelDataTypeFactory typeFactory =
+ new JavaTypeFactoryImpl();
+ RelOptCluster cluster = RelOptCluster.create(volcanoPlanner, new RexBuilder(typeFactory));
+ RelBuilder builder = RelFactories.LOGICAL_BUILDER.create(cluster, null);
+ RelNode initRel = customPlanFunction.apply(builder);
+
+ // convert to HyperGraph
+ HepProgram program = new HepProgramBuilder()
+ .addMatchOrder(HepMatchOrder.BOTTOM_UP)
+ .addRuleInstance(CoreRules.JOIN_TO_HYPER_GRAPH)
+ .build();
+ HepPlanner hepPlanner = new HepPlanner(program);
+ hepPlanner.setRoot(initRel);
+ RelNode hyperGraph = hepPlanner.findBestExp();
+ assert hyperGraph instanceof HyperGraph : "The root node must be a HyperGraph, "
+ + "please check the custom plan function. The root node now is:\n"
+ + RelOptUtil.toString(hyperGraph);
+
+ // start dphyp enumeration
+ DphypForTest dphyp =
+ new DphypForTest(
+ (HyperGraph) hyperGraph,
+ builder,
+ hyperGraph.getCluster().getMetadataQuery(),
+ 127);
+ dphyp.startEnumerateJoin();
+
+ // verify the execution results of each candidate plan enumerated by dphyp
+ for (RelNode candidatePlan : dphyp.candiateLists) {
+ for (RelOptRule enumerableRule : EnumerableRules.rules()) {
+ volcanoPlanner.addRule(enumerableRule);
+ }
+ for (RelOptRule calcRule : RelOptRules.CALC_RULES) {
+ volcanoPlanner.addRule(calcRule);
+ }
+ volcanoPlanner.removeRule(EnumerableRules.ENUMERABLE_PROJECT_RULE);
+
+ candidatePlan =
Review Comment:
I have considered making this function more general, but for the purpose of testing dphyp, it has some limitations:
- JoinToHyperGraphRule must be used.
- the Join nodes in the given plan must be as compact as possible so that they can be included in the same hypergraph.
- the given plan must have the Join as the root (to ensure that after the plan is converted to a HyperGraph, the root node is the HyperGraph for DphypForTest to call).
These limitations may not be convenient for other tests. Maybe in the future we can consider providing a similar method that is more general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org