You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by GitBox <gi...@apache.org> on 2020/06/12 18:22:26 UTC
[GitHub] [calcite] amaliujia opened a new pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
amaliujia opened a new pull request #2023:
URL: https://github.com/apache/calcite/pull/2023
…oopJoin (Rui Wang).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] amaliujia commented on a change in pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2023:
URL: https://github.com/apache/calcite/pull/2023#discussion_r439609801
##########
File path: core/src/main/java/org/apache/calcite/adapter/enumerable/EnumTraitsUtils.java
##########
@@ -148,4 +149,52 @@ private static boolean isCollationOnTrivialExpr(
return null;
}
}
+
+ // EnumerableNestedLoopJoin and EnumerableBatchNestedLoopJoin traits passdown shall only
+ // pass through collation to left input. It is because for EnumerableNestedLoopJoin always
+ // uses left input as the outer loop, thus only left input can preserve ordering.
+ // Push sort both to left and right inputs does not help right outer join. It's because in
+ // implementation, EnumerableNestedLoopJoin produces (null, right_unmatched) all together,
+ // which does not preserve ordering from right side.
+ static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraitsForNestedLoopJoin(
Review comment:
Pushed a commit to make HashJoin and Correlate reuse the code. Also rename this class to `EnumerableTraitUtils`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] rubenada commented on pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
Posted by GitBox <gi...@apache.org>.
rubenada commented on pull request #2023:
URL: https://github.com/apache/calcite/pull/2023#issuecomment-643588563
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] hsyuan closed pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
Posted by GitBox <gi...@apache.org>.
hsyuan closed pull request #2023:
URL: https://github.com/apache/calcite/pull/2023
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] hsyuan commented on a change in pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
Posted by GitBox <gi...@apache.org>.
hsyuan commented on a change in pull request #2023:
URL: https://github.com/apache/calcite/pull/2023#discussion_r439580414
##########
File path: core/src/main/java/org/apache/calcite/adapter/enumerable/EnumTraitsUtils.java
##########
@@ -148,4 +149,52 @@ private static boolean isCollationOnTrivialExpr(
return null;
}
}
+
+ // EnumerableNestedLoopJoin and EnumerableBatchNestedLoopJoin traits passdown shall only
+ // pass through collation to left input. It is because for EnumerableNestedLoopJoin always
+ // uses left input as the outer loop, thus only left input can preserve ordering.
+ // Push sort both to left and right inputs does not help right outer join. It's because in
+ // implementation, EnumerableNestedLoopJoin produces (null, right_unmatched) all together,
+ // which does not preserve ordering from right side.
+ static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraitsForNestedLoopJoin(
Review comment:
The class name is misleading. Should be EnumerableTraitUtil, Enum is a different thing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] amaliujia commented on a change in pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2023:
URL: https://github.com/apache/calcite/pull/2023#discussion_r439577088
##########
File path: core/src/test/java/org/apache/calcite/test/TopDownOptTest.java
##########
@@ -483,7 +476,6 @@
Query.create(sql)
.removeRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE)
- .removeRule(EnumerableRules.ENUMERABLE_BATCH_NESTED_LOOP_JOIN_RULE)
Review comment:
My bad. This rule is not added by default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] amaliujia commented on pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #2023:
URL: https://github.com/apache/calcite/pull/2023#issuecomment-643421126
R: @chunweilei @hsyuan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] hsyuan commented on a change in pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
Posted by GitBox <gi...@apache.org>.
hsyuan commented on a change in pull request #2023:
URL: https://github.com/apache/calcite/pull/2023#discussion_r439578915
##########
File path: core/src/main/java/org/apache/calcite/adapter/enumerable/EnumTraitsUtils.java
##########
@@ -148,4 +149,52 @@ private static boolean isCollationOnTrivialExpr(
return null;
}
}
+
+ // EnumerableNestedLoopJoin and EnumerableBatchNestedLoopJoin traits passdown shall only
+ // pass through collation to left input. It is because for EnumerableNestedLoopJoin always
+ // uses left input as the outer loop, thus only left input can preserve ordering.
+ // Push sort both to left and right inputs does not help right outer join. It's because in
+ // implementation, EnumerableNestedLoopJoin produces (null, right_unmatched) all together,
+ // which does not preserve ordering from right side.
+ static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraitsForNestedLoopJoin(
+ RelTraitSet required, JoinRelType joinType,
+ int leftInputFieldCount, RelTraitSet joinTraitSet) {
+ RelCollation collation = required.getCollation();
+ if (collation == null
+ || collation == RelCollations.EMPTY
+ || joinType == JoinRelType.FULL
+ || joinType == JoinRelType.RIGHT) {
+ return null;
+ }
+
+ for (RelFieldCollation fc : collation.getFieldCollations()) {
+ // If field collation belongs to right input: cannot push down collation.
+ if (fc.getFieldIndex() >= leftInputFieldCount) {
+ return null;
+ }
+ }
+
+ RelTraitSet passthroughTraitSet = joinTraitSet.replace(collation);
+ return Pair.of(passthroughTraitSet,
+ ImmutableList.of(
+ passthroughTraitSet,
+ passthroughTraitSet.replace(RelCollations.EMPTY)));
+ }
+
+ static Pair<RelTraitSet, List<RelTraitSet>> deriveTraitsForNestedLoopJoin(
+ RelTraitSet childTraits, int childId,
+ RelTraitSet joinTraitSet, RelTraitSet rightTraitSet) {
+ // should only derive traits (limited to collation for now) from left join input.
+ assert childId == 0;
+
+ RelCollation collation = childTraits.getCollation();
+ if (collation == null || collation == RelCollations.EMPTY) {
Review comment:
Add check
```
|| joinType == JoinRelType.FULL
|| joinType == JoinRelType.RIGHT
```
##########
File path: core/src/main/java/org/apache/calcite/adapter/enumerable/EnumTraitsUtils.java
##########
@@ -148,4 +149,52 @@ private static boolean isCollationOnTrivialExpr(
return null;
}
}
+
+ // EnumerableNestedLoopJoin and EnumerableBatchNestedLoopJoin traits passdown shall only
+ // pass through collation to left input. It is because for EnumerableNestedLoopJoin always
+ // uses left input as the outer loop, thus only left input can preserve ordering.
+ // Push sort both to left and right inputs does not help right outer join. It's because in
+ // implementation, EnumerableNestedLoopJoin produces (null, right_unmatched) all together,
+ // which does not preserve ordering from right side.
+ static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraitsForNestedLoopJoin(
Review comment:
HashJoin and Correlate can reuse it too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] rubenada commented on a change in pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
Posted by GitBox <gi...@apache.org>.
rubenada commented on a change in pull request #2023:
URL: https://github.com/apache/calcite/pull/2023#discussion_r439719710
##########
File path: core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableTraitsUtils.java
##########
@@ -148,4 +149,53 @@ private static boolean isCollationOnTrivialExpr(
return null;
}
}
+
+ // This function can be reused when a Join's traits pass-down shall only
Review comment:
minor detail: I think it would be better to use javadoc comment here (instead of line comment). Same for `deriveTraitsForJoin`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] amaliujia commented on a change in pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2023:
URL: https://github.com/apache/calcite/pull/2023#discussion_r439761102
##########
File path: core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableTraitsUtils.java
##########
@@ -148,4 +149,53 @@ private static boolean isCollationOnTrivialExpr(
return null;
}
}
+
+ // This function can be reused when a Join's traits pass-down shall only
Review comment:
Updated to javadoc. Thanks for you suggestion!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] amaliujia commented on a change in pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2023:
URL: https://github.com/apache/calcite/pull/2023#discussion_r439579435
##########
File path: core/src/main/java/org/apache/calcite/adapter/enumerable/EnumTraitsUtils.java
##########
@@ -148,4 +149,52 @@ private static boolean isCollationOnTrivialExpr(
return null;
}
}
+
+ // EnumerableNestedLoopJoin and EnumerableBatchNestedLoopJoin traits passdown shall only
+ // pass through collation to left input. It is because for EnumerableNestedLoopJoin always
+ // uses left input as the outer loop, thus only left input can preserve ordering.
+ // Push sort both to left and right inputs does not help right outer join. It's because in
+ // implementation, EnumerableNestedLoopJoin produces (null, right_unmatched) all together,
+ // which does not preserve ordering from right side.
+ static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraitsForNestedLoopJoin(
Review comment:
Nice catch. Will migrate them too to this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] amaliujia commented on a change in pull request #2023: [CALCITE-4057] Implement trait propagation for EnumerableBatchNestedL…
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2023:
URL: https://github.com/apache/calcite/pull/2023#discussion_r439577517
##########
File path: core/src/test/java/org/apache/calcite/test/TopDownOptTest.java
##########
@@ -693,6 +678,37 @@
.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE)
.check();
}
+
+ // push sort to left input
+ @Test void testBatchNestedLoopJoinLeftOuterJoinPushDownSort() {
Review comment:
Not adding too many tests for EnumerableBatchNestedLoopJoin because it reuses EnumerableNestedLoopJoin's code and those code was tested.
Add two test only to verify that trait propagation in EnumerableBatchNestedLoopJoin is working.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org