You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by GitBox <gi...@apache.org> on 2020/06/09 16:09:58 UTC

[GitHub] [calcite] chunweilei opened a new pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

chunweilei opened a new pull request #2011:
URL: https://github.com/apache/calcite/pull/2011


   JIRA: https://issues.apache.org/jira/browse/CALCITE-4016


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] chunweilei merged pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
chunweilei merged pull request #2011:
URL: https://github.com/apache/calcite/pull/2011


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] chunweilei commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
chunweilei commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r437829738



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,104 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFc = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFc)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  public static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       @hsyuan Instead of adding an interface, how about adding an EnumTraitUtils to hold these methods?  IMO, an interface seems not necessary since only Calc and Project should implement it. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] chunweilei commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
chunweilei commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r437119231



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,115 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Gets a shuttle to expand RexLocalRef.
+   */
+  public static RexShuttle getExpandShuttle(RexProgram rexProgram) {
+    return new RexShuttle() {
+      @Override public RexNode visitLocalRef(RexLocalRef localRef) {
+        return rexProgram.expandLocalRef(localRef);
+      }
+    };
+  }
+
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects,RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);

Review comment:
       I am not sure whether RelOptUtil is the best place to hold these methods. Any suggestion is appreciated.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] amaliujia commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r437592439



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,104 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFc = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFc)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  public static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       Depends on if we want move all most all the code to RelOptUtil, I feel like only `isCollationOnTrivialExpr` can be moved to RelOptUtil.
   
   At least `RelOptUtil.passThroughTraits` and `RelOptTuil. deriveTraits` seems to me, could encourage people write all `passThroughTraits` and `deriveTraits` into `RelOptUtil`, because names here are too generic.
   
   For example, I might ask, why not move implementations in Enumerable Joins to `RelOptUtil`, etc.
   

##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,104 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFc = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFc)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  public static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       Depends on if we want move all most all the code to RelOptUtil, I feel like only `isCollationOnTrivialExpr` can be moved to RelOptUtil.
   
   At least `RelOptUtil.passThroughTraits` and `RelOptTuil. deriveTraits`, seems to me, could encourage people write all `passThroughTraits` and `deriveTraits` into `RelOptUtil`, because names here are too generic.
   
   For example, I might ask, why not move implementations in Enumerable Joins to `RelOptUtil`, etc.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] hsyuan commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
hsyuan commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r438317367



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,104 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFc = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFc)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  public static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       Sounds fare. Be sure to make it package level class, and mark it for internal use.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] amaliujia commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r437588277



##########
File path: core/src/test/java/org/apache/calcite/test/TopDownOptTest.java
##########
@@ -573,6 +575,112 @@
         .removeRule(EnumerableRules.ENUMERABLE_SORT_RULE)
         .check();
   }
+
+  // test if "order by mgr desc nulls last" can be pushed through the calc ("select mgr").
+  @Test void testSortCalc() {
+    final String sql = "select mgr from sales.emp order by mgr desc nulls last";
+    Query.create(sql)
+        .addRule(ProjectToCalcRule.INSTANCE)
+        .addRule(EnumerableRules.ENUMERABLE_CALC_RULE)
+        .removeRule(EnumerableRules.ENUMERABLE_SORT_RULE)
+        .removeRule(EnumerableRules.ENUMERABLE_PROJECT_RULE)
+        .check();
+  }
+
+  // test that Sort cannot push through calc because of non-trival call
+  // (e.g. RexCall(sal * -1)). In this example, the reason is that "sal * -1"
+  // creates opposite ordering if Sort is pushed down.
+  @Test void testSortCalcOnRexCall() {
+    final String sql = "select ename, sal * -1 as sal, mgr from\n"
+        + "sales.emp order by ename desc, sal desc, mgr desc nulls last";
+    Query.create(sql)
+        .addRule(ProjectToCalcRule.INSTANCE)
+        .addRule(EnumerableRules.ENUMERABLE_CALC_RULE)
+        .removeRule(EnumerableRules.ENUMERABLE_SORT_RULE)
+        .removeRule(EnumerableRules.ENUMERABLE_PROJECT_RULE)
+        .check();
+  }
+
+  // test that Sort can push through calc when cast is monotonic.
+  @Test void testSortCalcWhenCastLeadingToMonotonic() {
+    final String sql = "select deptno from sales.emp order by cast(deptno as float) desc";

Review comment:
       I think I made a mistake on such tests. To improve readability, it's better to do 
   
   `select cast(deptno as float) from sales.emp order by  deptno desc`.
   
   But these two queries are testing the same thing.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] chunweilei commented on pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
chunweilei commented on pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#issuecomment-643146964


   Thank you for your review, @amaliujia @hsyuan .


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] amaliujia commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r439053221



##########
File path: core/src/main/java/org/apache/calcite/adapter/enumerable/EnumTraitsUtils.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.adapter.enumerable;
+
+import org.apache.calcite.linq4j.Ord;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelCollations;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCallBinding;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.validate.SqlMonotonicity;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.mapping.MappingType;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+
+import org.apiguardian.api.API;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * Utilities for traits propagation.
+ */
+@API(since = "1.24", status = API.Status.INTERNAL)
+class EnumTraitsUtils {
+
+  private EnumTraitsUtils() {}
+
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFieldCollation = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFieldCollation)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(
+      RelTraitSet required,
+      List<RexNode> exps,
+      RelDataType inputRowType,
+      RelDataTypeFactory typeFactory,
+      RelTraitSet currentTraits) {
+    final RelCollation collation = required.getCollation();
+    if (collation == null || collation == RelCollations.EMPTY) {
+      return null;
+    }
+
+    final Mappings.TargetMapping map =
+        RelOptUtil.permutationIgnoreCast(
+            exps, inputRowType);
+
+    if (collation.getFieldCollations().stream().anyMatch(
+        rc -> !isCollationOnTrivialExpr(exps, typeFactory,
+            map, rc, true))) {
+      return null;
+    }
+
+    final RelCollation newCollation = collation.apply(map);
+    return Pair.of(currentTraits.replace(collation),
+        ImmutableList.of(currentTraits.replace(newCollation)));
+  }
+
+  static Pair<RelTraitSet, List<RelTraitSet>> deriveTraits(

Review comment:
       Same for deriveTraits

##########
File path: core/src/main/java/org/apache/calcite/adapter/enumerable/EnumTraitsUtils.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.adapter.enumerable;
+
+import org.apache.calcite.linq4j.Ord;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelCollations;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCallBinding;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.validate.SqlMonotonicity;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.mapping.MappingType;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+
+import org.apiguardian.api.API;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * Utilities for traits propagation.
+ */
+@API(since = "1.24", status = API.Status.INTERNAL)
+class EnumTraitsUtils {
+
+  private EnumTraitsUtils() {}
+
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFieldCollation = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFieldCollation)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       Can you rename it to `passThroughTraitsProject`? I will also move some code for NestedLoopJoin here, so I can use `passThroughTraitsNestedLoopJoin` to differentiate.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] amaliujia commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r439053787



##########
File path: core/src/test/resources/org/apache/calcite/test/TopDownOptTest.xml
##########
@@ -441,223 +441,223 @@ EnumerableMergeJoin(condition=[AND(=($1, $4), =($0, $3))], joinType=[inner])
   EnumerableSort(sort0=[$1], sort1=[$0], dir0=[ASC], dir1=[ASC])
     EnumerableTableScan(table=[[CATALOG, SALES, BONUS]])
 ]]>
-    </Resource>
-  </TestCase>
-  <TestCase name="testSortAggPartialKey">
-    <Resource name="sql">
-      <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
+        </Resource>
+    </TestCase>
+    <TestCase name="testSortAggPartialKey">
+        <Resource name="sql">
+            <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
 group by mgr,deptno,comm
 order by comm desc nulls last, deptno nulls first]]>
-    </Resource>
-    <Resource name="planBefore">
-      <![CDATA[
+        </Resource>
+        <Resource name="planBefore">
+            <![CDATA[
 LogicalSort(sort0=[$2], sort1=[$1], dir0=[DESC-nulls-last], dir1=[ASC-nulls-first])
   LogicalAggregate(group=[{0, 1, 2}], EXPR$3=[COUNT()])
     LogicalProject(MGR=[$3], DEPTNO=[$7], COMM=[$6])
       LogicalTableScan(table=[[CATALOG, SALES, EMP]])
 ]]>
-    </Resource>
-    <Resource name="planAfter">
-      <![CDATA[
+        </Resource>
+        <Resource name="planAfter">
+            <![CDATA[

Review comment:
       +1. Introduce lost of changes just due to indention seems not good.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] chunweilei commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
chunweilei commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r439168431



##########
File path: core/src/test/resources/org/apache/calcite/test/TopDownOptTest.xml
##########
@@ -441,223 +441,223 @@ EnumerableMergeJoin(condition=[AND(=($1, $4), =($0, $3))], joinType=[inner])
   EnumerableSort(sort0=[$1], sort1=[$0], dir0=[ASC], dir1=[ASC])
     EnumerableTableScan(table=[[CATALOG, SALES, BONUS]])
 ]]>
-    </Resource>
-  </TestCase>
-  <TestCase name="testSortAggPartialKey">
-    <Resource name="sql">
-      <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
+        </Resource>
+    </TestCase>
+    <TestCase name="testSortAggPartialKey">
+        <Resource name="sql">
+            <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
 group by mgr,deptno,comm
 order by comm desc nulls last, deptno nulls first]]>
-    </Resource>
-    <Resource name="planBefore">
-      <![CDATA[
+        </Resource>
+        <Resource name="planBefore">
+            <![CDATA[
 LogicalSort(sort0=[$2], sort1=[$1], dir0=[DESC-nulls-last], dir1=[ASC-nulls-first])
   LogicalAggregate(group=[{0, 1, 2}], EXPR$3=[COUNT()])
     LogicalProject(MGR=[$3], DEPTNO=[$7], COMM=[$6])
       LogicalTableScan(table=[[CATALOG, SALES, EMP]])
 ]]>
-    </Resource>
-    <Resource name="planAfter">
-      <![CDATA[
+        </Resource>
+        <Resource name="planAfter">
+            <![CDATA[

Review comment:
       Oops, will do.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] amaliujia commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r437586908



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,115 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Gets a shuttle to expand RexLocalRef.
+   */
+  public static RexShuttle getExpandShuttle(RexProgram rexProgram) {
+    return new RexShuttle() {
+      @Override public RexNode visitLocalRef(RexLocalRef localRef) {
+        return rexProgram.expandLocalRef(localRef);
+      }
+    };
+  }
+
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects,RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);

Review comment:
       I think the part of construct pairs of traits for `passThroughTraits` and `dervieTratis` are better kept in Enumerable nodes, at least for better readbility.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] amaliujia commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r437586908



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,115 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Gets a shuttle to expand RexLocalRef.
+   */
+  public static RexShuttle getExpandShuttle(RexProgram rexProgram) {
+    return new RexShuttle() {
+      @Override public RexNode visitLocalRef(RexLocalRef localRef) {
+        return rexProgram.expandLocalRef(localRef);
+      }
+    };
+  }
+
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects,RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);

Review comment:
       I think the part of construct pairs of traits for `passThroughTraits` and `dervieTratis` are better kept in Enumerable nodes, at least for better readability.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] chunweilei commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
chunweilei commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r438514694



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,104 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFc = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFc)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  public static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       Updated.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] amaliujia commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r437592439



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,104 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFc = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFc)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  public static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       Depends on if we want move all most all the code to RelOptUtil, I feel like only `isCollationOnTrivialExpr` can be moved to RelOptUtil.
   
   At least `RelOptUtil.passThroughTraits` and `RelOptTuil. deriveTraits`, seems to me, could encourage people write all `passThroughTraits` and `deriveTraits` into `RelOptUtil`, because names here are too generic.
   
   For example, I might ask, why not move implementations in Enumerable Joins to `RelOptUtil`, etc. And will they become a part of `RelOptUtil.passThroughTraits` and `RelOptTuil. deriveTraits`?
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] chunweilei commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
chunweilei commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r439168374



##########
File path: core/src/main/java/org/apache/calcite/adapter/enumerable/EnumTraitsUtils.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.adapter.enumerable;
+
+import org.apache.calcite.linq4j.Ord;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelCollations;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCallBinding;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.validate.SqlMonotonicity;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.mapping.MappingType;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+
+import org.apiguardian.api.API;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * Utilities for traits propagation.
+ */
+@API(since = "1.24", status = API.Status.INTERNAL)
+class EnumTraitsUtils {
+
+  private EnumTraitsUtils() {}
+
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFieldCollation = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFieldCollation)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       Sure.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] hsyuan commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
hsyuan commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r439162118



##########
File path: core/src/test/resources/org/apache/calcite/test/TopDownOptTest.xml
##########
@@ -441,223 +441,223 @@ EnumerableMergeJoin(condition=[AND(=($1, $4), =($0, $3))], joinType=[inner])
   EnumerableSort(sort0=[$1], sort1=[$0], dir0=[ASC], dir1=[ASC])
     EnumerableTableScan(table=[[CATALOG, SALES, BONUS]])
 ]]>
-    </Resource>
-  </TestCase>
-  <TestCase name="testSortAggPartialKey">
-    <Resource name="sql">
-      <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
+        </Resource>
+    </TestCase>
+    <TestCase name="testSortAggPartialKey">
+        <Resource name="sql">
+            <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
 group by mgr,deptno,comm
 order by comm desc nulls last, deptno nulls first]]>
-    </Resource>
-    <Resource name="planBefore">
-      <![CDATA[
+        </Resource>
+        <Resource name="planBefore">
+            <![CDATA[
 LogicalSort(sort0=[$2], sort1=[$1], dir0=[DESC-nulls-last], dir1=[ASC-nulls-first])
   LogicalAggregate(group=[{0, 1, 2}], EXPR$3=[COUNT()])
     LogicalProject(MGR=[$3], DEPTNO=[$7], COMM=[$6])
       LogicalTableScan(table=[[CATALOG, SALES, EMP]])
 ]]>
-    </Resource>
-    <Resource name="planAfter">
-      <![CDATA[
+        </Resource>
+        <Resource name="planAfter">
+            <![CDATA[

Review comment:
       thanks!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] hsyuan commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
hsyuan commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r437658417



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,104 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFc = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFc)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  public static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       I had the same feeling. How about add a package level internal interface for EnumerableProject and EnumerableCalc with default implementation so that both operators just need to implement the interface?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] chunweilei commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
chunweilei commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r437830079



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,104 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFc = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFc)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  public static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       @amaliujia We could change the name to clarify what it is doing.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] amaliujia commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r438296691



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3766,6 +3771,104 @@ private static void fix(List<RexNode> operands, int before, int after) {
     }
   }
 
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFc = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFc)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  public static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       Sure. Either have a EnumTraitUtils (to host as much code to share for traits propagation) or rename it is fine.
   
   I find there is another case to share trait propagation code: https://jira.apache.org/jira/browse/CALCITE-4057. So after this PR to build a code sharing place, we can also apply it to  CALCITE-4057.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] hsyuan commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
hsyuan commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r439153234



##########
File path: core/src/test/resources/org/apache/calcite/test/TopDownOptTest.xml
##########
@@ -441,223 +441,223 @@ EnumerableMergeJoin(condition=[AND(=($1, $4), =($0, $3))], joinType=[inner])
   EnumerableSort(sort0=[$1], sort1=[$0], dir0=[ASC], dir1=[ASC])
     EnumerableTableScan(table=[[CATALOG, SALES, BONUS]])
 ]]>
-    </Resource>
-  </TestCase>
-  <TestCase name="testSortAggPartialKey">
-    <Resource name="sql">
-      <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
+        </Resource>
+    </TestCase>
+    <TestCase name="testSortAggPartialKey">
+        <Resource name="sql">
+            <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
 group by mgr,deptno,comm
 order by comm desc nulls last, deptno nulls first]]>
-    </Resource>
-    <Resource name="planBefore">
-      <![CDATA[
+        </Resource>
+        <Resource name="planBefore">
+            <![CDATA[
 LogicalSort(sort0=[$2], sort1=[$1], dir0=[DESC-nulls-last], dir1=[ASC-nulls-first])
   LogicalAggregate(group=[{0, 1, 2}], EXPR$3=[COUNT()])
     LogicalProject(MGR=[$3], DEPTNO=[$7], COMM=[$6])
       LogicalTableScan(table=[[CATALOG, SALES, EMP]])
 ]]>
-    </Resource>
-    <Resource name="planAfter">
-      <![CDATA[
+        </Resource>
+        <Resource name="planAfter">
+            <![CDATA[

Review comment:
       BTW, there is another JIRA CALCITE-4027, do you mind taking it, @amaliujia ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] amaliujia commented on pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#issuecomment-641460202


   cc @hsyuan to raise attention on this PR.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] amaliujia commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r439053179



##########
File path: core/src/main/java/org/apache/calcite/adapter/enumerable/EnumTraitsUtils.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.adapter.enumerable;
+
+import org.apache.calcite.linq4j.Ord;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelCollations;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexCallBinding;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.validate.SqlMonotonicity;
+import org.apache.calcite.util.Pair;
+import org.apache.calcite.util.mapping.MappingType;
+import org.apache.calcite.util.mapping.Mappings;
+
+import com.google.common.collect.ImmutableList;
+
+import org.apiguardian.api.API;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * Utilities for traits propagation.
+ */
+@API(since = "1.24", status = API.Status.INTERNAL)
+class EnumTraitsUtils {
+
+  private EnumTraitsUtils() {}
+
+  /**
+   * Determine whether there is mapping between project input and output fields.
+   * Bail out if sort relies on non-trivial expressions.
+   */
+  private static boolean isCollationOnTrivialExpr(
+      List<RexNode> projects, RelDataTypeFactory typeFactory,
+      Mappings.TargetMapping map, RelFieldCollation fc, boolean passDown) {
+    final int index = fc.getFieldIndex();
+    int target = map.getTargetOpt(index);
+    if (target < 0) {
+      return false;
+    }
+
+    final RexNode node = passDown ? projects.get(index) : projects.get(target);
+    if (node.isA(SqlKind.CAST)) {
+      // Check whether it is a monotonic preserving cast
+      final RexCall cast = (RexCall) node;
+      RelFieldCollation newFieldCollation = Objects.requireNonNull(RexUtil.apply(map, fc));
+      final RexCallBinding binding =
+          RexCallBinding.create(typeFactory, cast,
+              ImmutableList.of(RelCollations.of(newFieldCollation)));
+      if (cast.getOperator().getMonotonicity(binding)
+          == SqlMonotonicity.NOT_MONOTONIC) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  static Pair<RelTraitSet, List<RelTraitSet>> passThroughTraits(

Review comment:
       Can you rename it to `passThroughTraitsProject`. I will also move some code for NestedLoopJoin here, so I can use `passThroughTraitsNestedLoopJoin` to differentiate.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] chunweilei commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
chunweilei commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r437827477



##########
File path: core/src/test/java/org/apache/calcite/test/TopDownOptTest.java
##########
@@ -573,6 +575,112 @@
         .removeRule(EnumerableRules.ENUMERABLE_SORT_RULE)
         .check();
   }
+
+  // test if "order by mgr desc nulls last" can be pushed through the calc ("select mgr").
+  @Test void testSortCalc() {
+    final String sql = "select mgr from sales.emp order by mgr desc nulls last";
+    Query.create(sql)
+        .addRule(ProjectToCalcRule.INSTANCE)
+        .addRule(EnumerableRules.ENUMERABLE_CALC_RULE)
+        .removeRule(EnumerableRules.ENUMERABLE_SORT_RULE)
+        .removeRule(EnumerableRules.ENUMERABLE_PROJECT_RULE)
+        .check();
+  }
+
+  // test that Sort cannot push through calc because of non-trival call
+  // (e.g. RexCall(sal * -1)). In this example, the reason is that "sal * -1"
+  // creates opposite ordering if Sort is pushed down.
+  @Test void testSortCalcOnRexCall() {
+    final String sql = "select ename, sal * -1 as sal, mgr from\n"
+        + "sales.emp order by ename desc, sal desc, mgr desc nulls last";
+    Query.create(sql)
+        .addRule(ProjectToCalcRule.INSTANCE)
+        .addRule(EnumerableRules.ENUMERABLE_CALC_RULE)
+        .removeRule(EnumerableRules.ENUMERABLE_SORT_RULE)
+        .removeRule(EnumerableRules.ENUMERABLE_PROJECT_RULE)
+        .check();
+  }
+
+  // test that Sort can push through calc when cast is monotonic.
+  @Test void testSortCalcWhenCastLeadingToMonotonic() {
+    final String sql = "select deptno from sales.emp order by cast(deptno as float) desc";

Review comment:
       Makes sense.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] hsyuan commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
hsyuan commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r438518362



##########
File path: core/src/test/resources/org/apache/calcite/test/TopDownOptTest.xml
##########
@@ -441,223 +441,223 @@ EnumerableMergeJoin(condition=[AND(=($1, $4), =($0, $3))], joinType=[inner])
   EnumerableSort(sort0=[$1], sort1=[$0], dir0=[ASC], dir1=[ASC])
     EnumerableTableScan(table=[[CATALOG, SALES, BONUS]])
 ]]>
-    </Resource>
-  </TestCase>
-  <TestCase name="testSortAggPartialKey">
-    <Resource name="sql">
-      <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
+        </Resource>
+    </TestCase>
+    <TestCase name="testSortAggPartialKey">
+        <Resource name="sql">
+            <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
 group by mgr,deptno,comm
 order by comm desc nulls last, deptno nulls first]]>
-    </Resource>
-    <Resource name="planBefore">
-      <![CDATA[
+        </Resource>
+        <Resource name="planBefore">
+            <![CDATA[
 LogicalSort(sort0=[$2], sort1=[$1], dir0=[DESC-nulls-last], dir1=[ASC-nulls-first])
   LogicalAggregate(group=[{0, 1, 2}], EXPR$3=[COUNT()])
     LogicalProject(MGR=[$3], DEPTNO=[$7], COMM=[$6])
       LogicalTableScan(table=[[CATALOG, SALES, EMP]])
 ]]>
-    </Resource>
-    <Resource name="planAfter">
-      <![CDATA[
+        </Resource>
+        <Resource name="planAfter">
+            <![CDATA[

Review comment:
       hmm.. Looks like we have different style settings.
   Can you remove these diffs?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] amaliujia commented on a change in pull request #2011: [CALCITE-4016] Support trait propagation for EnumerableCalc

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #2011:
URL: https://github.com/apache/calcite/pull/2011#discussion_r439161388



##########
File path: core/src/test/resources/org/apache/calcite/test/TopDownOptTest.xml
##########
@@ -441,223 +441,223 @@ EnumerableMergeJoin(condition=[AND(=($1, $4), =($0, $3))], joinType=[inner])
   EnumerableSort(sort0=[$1], sort1=[$0], dir0=[ASC], dir1=[ASC])
     EnumerableTableScan(table=[[CATALOG, SALES, BONUS]])
 ]]>
-    </Resource>
-  </TestCase>
-  <TestCase name="testSortAggPartialKey">
-    <Resource name="sql">
-      <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
+        </Resource>
+    </TestCase>
+    <TestCase name="testSortAggPartialKey">
+        <Resource name="sql">
+            <![CDATA[select mgr,deptno,comm,count(*) from sales.emp
 group by mgr,deptno,comm
 order by comm desc nulls last, deptno nulls first]]>
-    </Resource>
-    <Resource name="planBefore">
-      <![CDATA[
+        </Resource>
+        <Resource name="planBefore">
+            <![CDATA[
 LogicalSort(sort0=[$2], sort1=[$1], dir0=[DESC-nulls-last], dir1=[ASC-nulls-first])
   LogicalAggregate(group=[{0, 1, 2}], EXPR$3=[COUNT()])
     LogicalProject(MGR=[$3], DEPTNO=[$7], COMM=[$6])
       LogicalTableScan(table=[[CATALOG, SALES, EMP]])
 ]]>
-    </Resource>
-    <Resource name="planAfter">
-      <![CDATA[
+        </Resource>
+        <Resource name="planAfter">
+            <![CDATA[

Review comment:
       Sure. Will take a look.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org