You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by GitBox <gi...@apache.org> on 2020/09/22 11:15:01 UTC

[GitHub] [calcite] yanlin-Lynn opened a new pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

yanlin-Lynn opened a new pull request #2163:
URL: https://github.com/apache/calcite/pull/2163


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] yanlin-Lynn merged pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

Posted by GitBox <gi...@apache.org>.
yanlin-Lynn merged pull request #2163:
URL: https://github.com/apache/calcite/pull/2163


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] yanlin-Lynn commented on a change in pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

Posted by GitBox <gi...@apache.org>.
yanlin-Lynn commented on a change in pull request #2163:
URL: https://github.com/apache/calcite/pull/2163#discussion_r516402810



##########
File path: core/src/main/java/org/apache/calcite/rel/metadata/RelMdExpressionLineage.java
##########
@@ -407,6 +409,35 @@ protected RelMdExpressionLineage() {}
     return mq.getExpressionLineage(rel.getInput(), outputExpression);
   }
 
+  /**
+   * Expression lineage from Calc.
+   */
+  public Set<RexNode> getExpressionLineage(Calc calc,
+      RelMetadataQuery mq, RexNode outputExpression) {
+    final RelNode input = calc.getInput();
+    final RexBuilder rexBuilder = calc.getCluster().getRexBuilder();
+    // Extract input fields referenced by expression
+    final ImmutableBitSet inputFieldsUsed = extractInputRefs(outputExpression);
+
+    // Infer column origin expressions for given references
+    final Map<RexInputRef, Set<RexNode>> mapping = new LinkedHashMap<>();
+    Pair<ImmutableList<RexNode>, ImmutableList<RexNode>> calcFilterAndProjects =
+        calc.getProgram().split();

Review comment:
       OK, I'll update 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] yanlin-Lynn commented on pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

Posted by GitBox <gi...@apache.org>.
yanlin-Lynn commented on pull request #2163:
URL: https://github.com/apache/calcite/pull/2163#issuecomment-708937963


   > Could you use `RexProgram.split`? It is very similar to your code.
   
   Yes, it works for me. Didn't know that, thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] yanlin-Lynn commented on a change in pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

Posted by GitBox <gi...@apache.org>.
yanlin-Lynn commented on a change in pull request #2163:
URL: https://github.com/apache/calcite/pull/2163#discussion_r500925267



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3607,6 +3607,25 @@ public int size() {
     return relBuilder.build();
   }
 
+  /** Explain filtering condition and projections for Calc. */
+  public static Pair<RexNode, List<RexNode>> explainCalc(Calc calc) {

Review comment:
       Sure, I'll update




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] vlsi commented on a change in pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

Posted by GitBox <gi...@apache.org>.
vlsi commented on a change in pull request #2163:
URL: https://github.com/apache/calcite/pull/2163#discussion_r492663176



##########
File path: core/src/test/java/org/apache/calcite/test/RelMetadataTest.java
##########
@@ -2506,6 +2506,30 @@ private void checkPredicates(RelOptCluster cluster, RelOptTable empTable,
     assertNull(r);
   }
 
+  @Test void testExpressionLineageCalc() {
+    final RelNode rel = convertSql("select sal from (\n"
+        + " select * from emp) "
+        + "where deptno = 10");
+    final RelNode tableRel = convertSql("select * from emp");
+    final HepProgramBuilder programBuilder = HepProgram.builder();
+    programBuilder.addRuleInstance(CoreRules.PROJECT_TO_CALC);
+    programBuilder.addRuleInstance(CoreRules.FILTER_TO_CALC);
+    programBuilder.addRuleInstance(CoreRules.CALC_MERGE);
+    final HepPlanner planner = new HepPlanner(programBuilder.build());
+    planner.setRoot(rel);
+    final RelNode optimizedRel = planner.findBestExp();
+    final RelMetadataQuery mq = optimizedRel.getCluster().getMetadataQuery();
+
+    final RexNode ref = RexInputRef.of(0, optimizedRel.getRowType().getFieldList());
+    final Set<RexNode> r = mq.getExpressionLineage(optimizedRel, ref);
+
+    final String inputRef = RexInputRef.of(5, tableRel.getRowType().getFieldList()).toString();
+    assertThat(r.size(), is(1));
+    final String resultString = r.iterator().next().toString();
+    assertThat(resultString, startsWith(EMP_QNAME.toString()));
+    assertThat(resultString, endsWith(inputRef));

Review comment:
       I'm afraid it is obscure what is the expected output and why is that expected. Do you think you can convert the expected value to a single string with a corresponding message that clarifies why the value is expected?
   
   For instance, when testing `planning rules` we use full explain plan for comparison rather than adding lots assertions.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] danny0405 commented on a change in pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #2163:
URL: https://github.com/apache/calcite/pull/2163#discussion_r515829778



##########
File path: core/src/main/java/org/apache/calcite/rel/metadata/RelMdExpressionLineage.java
##########
@@ -407,6 +409,35 @@ protected RelMdExpressionLineage() {}
     return mq.getExpressionLineage(rel.getInput(), outputExpression);
   }
 
+  /**
+   * Expression lineage from Calc.
+   */
+  public Set<RexNode> getExpressionLineage(Calc calc,
+      RelMetadataQuery mq, RexNode outputExpression) {
+    final RelNode input = calc.getInput();
+    final RexBuilder rexBuilder = calc.getCluster().getRexBuilder();
+    // Extract input fields referenced by expression
+    final ImmutableBitSet inputFieldsUsed = extractInputRefs(outputExpression);
+
+    // Infer column origin expressions for given references
+    final Map<RexInputRef, Set<RexNode>> mapping = new LinkedHashMap<>();
+    Pair<ImmutableList<RexNode>, ImmutableList<RexNode>> calcFilterAndProjects =
+        calc.getProgram().split();

Review comment:
       `calcFilterAndProjects ` -> `calcProjectsAndFilter` ? Because projects are the key part.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] vlsi commented on a change in pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

Posted by GitBox <gi...@apache.org>.
vlsi commented on a change in pull request #2163:
URL: https://github.com/apache/calcite/pull/2163#discussion_r492657451



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3607,6 +3607,25 @@ public int size() {
     return relBuilder.build();
   }
 
+  /** Explain filtering condition and projections for Calc. */
+  public static Pair<RexNode, List<RexNode>> explainCalc(Calc calc) {

Review comment:
       Could you please elaborate on what does `explainCalc` mean?
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] julianhyde commented on a change in pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

Posted by GitBox <gi...@apache.org>.
julianhyde commented on a change in pull request #2163:
URL: https://github.com/apache/calcite/pull/2163#discussion_r493715390



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3607,6 +3607,25 @@ public int size() {
     return relBuilder.build();
   }
 
+  /** Explain filtering condition and projections for Calc. */
+  public static Pair<RexNode, List<RexNode>> explainCalc(Calc calc) {

Review comment:
       Maybe put a simple (non-trivial) example in the javadoc. Expressions, common sub-expressions, filter, some output fields based on constants, some input fields pruned from the output.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] julianhyde commented on pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

Posted by GitBox <gi...@apache.org>.
julianhyde commented on pull request #2163:
URL: https://github.com/apache/calcite/pull/2163#issuecomment-705179885


   Could you use `RexProgram.split`? It is very similar to your code.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] vlsi commented on a change in pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

Posted by GitBox <gi...@apache.org>.
vlsi commented on a change in pull request #2163:
URL: https://github.com/apache/calcite/pull/2163#discussion_r492657451



##########
File path: core/src/main/java/org/apache/calcite/plan/RelOptUtil.java
##########
@@ -3607,6 +3607,25 @@ public int size() {
     return relBuilder.build();
   }
 
+  /** Explain filtering condition and projections for Calc. */
+  public static Pair<RexNode, List<RexNode>> explainCalc(Calc calc) {

Review comment:
       Could you please elaborate on what does `explainCalc` mean?
   

##########
File path: core/src/test/java/org/apache/calcite/test/RelMetadataTest.java
##########
@@ -2506,6 +2506,30 @@ private void checkPredicates(RelOptCluster cluster, RelOptTable empTable,
     assertNull(r);
   }
 
+  @Test void testExpressionLineageCalc() {
+    final RelNode rel = convertSql("select sal from (\n"
+        + " select * from emp) "
+        + "where deptno = 10");
+    final RelNode tableRel = convertSql("select * from emp");
+    final HepProgramBuilder programBuilder = HepProgram.builder();
+    programBuilder.addRuleInstance(CoreRules.PROJECT_TO_CALC);
+    programBuilder.addRuleInstance(CoreRules.FILTER_TO_CALC);
+    programBuilder.addRuleInstance(CoreRules.CALC_MERGE);
+    final HepPlanner planner = new HepPlanner(programBuilder.build());
+    planner.setRoot(rel);
+    final RelNode optimizedRel = planner.findBestExp();
+    final RelMetadataQuery mq = optimizedRel.getCluster().getMetadataQuery();
+
+    final RexNode ref = RexInputRef.of(0, optimizedRel.getRowType().getFieldList());
+    final Set<RexNode> r = mq.getExpressionLineage(optimizedRel, ref);
+
+    final String inputRef = RexInputRef.of(5, tableRel.getRowType().getFieldList()).toString();
+    assertThat(r.size(), is(1));
+    final String resultString = r.iterator().next().toString();
+    assertThat(resultString, startsWith(EMP_QNAME.toString()));
+    assertThat(resultString, endsWith(inputRef));

Review comment:
       I'm afraid it is obscure what is the expected output and why is that expected. Do you think you can convert the expected value to a single string with a corresponding message that clarifies why the value is expected?
   
   For instance, when testing `planning rules` we use full explain plan for comparison rather than adding lots assertions.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [calcite] yanlin-Lynn commented on a change in pull request #2163: [CALCITE-4273] Support get expression lineage for Calc

Posted by GitBox <gi...@apache.org>.
yanlin-Lynn commented on a change in pull request #2163:
URL: https://github.com/apache/calcite/pull/2163#discussion_r500926077



##########
File path: core/src/test/java/org/apache/calcite/test/RelMetadataTest.java
##########
@@ -2506,6 +2506,30 @@ private void checkPredicates(RelOptCluster cluster, RelOptTable empTable,
     assertNull(r);
   }
 
+  @Test void testExpressionLineageCalc() {
+    final RelNode rel = convertSql("select sal from (\n"
+        + " select * from emp) "
+        + "where deptno = 10");
+    final RelNode tableRel = convertSql("select * from emp");
+    final HepProgramBuilder programBuilder = HepProgram.builder();
+    programBuilder.addRuleInstance(CoreRules.PROJECT_TO_CALC);
+    programBuilder.addRuleInstance(CoreRules.FILTER_TO_CALC);
+    programBuilder.addRuleInstance(CoreRules.CALC_MERGE);
+    final HepPlanner planner = new HepPlanner(programBuilder.build());
+    planner.setRoot(rel);
+    final RelNode optimizedRel = planner.findBestExp();
+    final RelMetadataQuery mq = optimizedRel.getCluster().getMetadataQuery();
+
+    final RexNode ref = RexInputRef.of(0, optimizedRel.getRowType().getFieldList());
+    final Set<RexNode> r = mq.getExpressionLineage(optimizedRel, ref);
+
+    final String inputRef = RexInputRef.of(5, tableRel.getRowType().getFieldList()).toString();
+    assertThat(r.size(), is(1));
+    final String resultString = r.iterator().next().toString();
+    assertThat(resultString, startsWith(EMP_QNAME.toString()));
+    assertThat(resultString, endsWith(inputRef));

Review comment:
       This case just follow the style of other cases in this class.
   To make it simple to read, I'll add a check for the full string result in the end of case.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org