[jira] [Comment Edited] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule

["Vladimir Sitnikov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16759498#comment-16759498 ] 

Vladimir Sitnikov edited comment on CALCITE-2223 at 2/3/19 7:28 PM:
--------------------------------------------------------------------

Cost for DrillFilterRel(DrillJoinRel( is smaller than the one of DrillJoinRel since it thinks it doesn't need to build and probe the hashtable.
Input sizes are estimated as 1 row and 10'000 rows.

That does not create the problem alone, however Drill uses separate planning phase for "join planning" which causes problem.

In other words, it does pick Filter(Cartesian(Cartesian(..)) in the first Volcano phase, then it can't be improved further since MultiJoin sees just a couple of cartesian joins, so the ordering makes no difference.

Adding {{Math.max(1, }} to DrillJoinRel does heal this test, however I'm not sure what is the real cost model there.
{code:java}  protected RelOptCost computeHashJoinCost(RelOptPlanner planner, RelMetadataQuery mq) {
      return computeHashJoinCostWithKeySize(planner, Math.max(1, this.getLeftKeys().size()), mq);
  }{code}


was (Author: vladimirsitnikov):
Cost for DrillFilterRel(DrillJoinRel( is smaller than the one of DrillJoinRel since it thinks it doesn't need to build and probe the hashtable.
Input sizes are estimated as 1 row and 10'000 rows.

That does not create the problem alone, however Drill uses separate planning phase for "join planning" which causes problem.

In other words, it does pick Filter(Cartesian(Cartesian(..)) in the first Volcano phase, then it can't be improved further since MultiJoin sees just a couple of cartesian joins, so the ordering makes no difference.

Adding {{Math.max(1, }} to DrillJoinRel does heal this test, however I'm not sure whats the real cost model there.
{code:java}  protected RelOptCost computeHashJoinCost(RelOptPlanner planner, RelMetadataQuery mq) {
      return computeHashJoinCostWithKeySize(planner, Math.max(1, this.getLeftKeys().size()), mq);
  }{code}

> ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
> -----------------------------------------------------------------------------------------
>
>                 Key: CALCITE-2223
>                 URL: https://issues.apache.org/jira/browse/CALCITE-2223
>             Project: Calcite
>          Issue Type: Bug
>            Reporter: Volodymyr Vysotskyi
>            Assignee: Julian Hyde
>            Priority: Critical
>         Attachments: TestLimitWithExchanges_testPushLimitPastUnionExchange.png, heap_overview.png, provenance_contents.png
>
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
>  A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
>   @Test public void testOomProjectMergeRule() {
>     RelBuilder relBuilder = RelBuilder.create(RelBuilderTest.config().build());
>     RelNode relNode = relBuilder
>         .values(new String[]{"f"}, "1")
>         .project(
>             relBuilder.alias(
>                 relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
>                 "f"))
>         .project(
>             relBuilder.alias(
>                 relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
>                 "f0"),
>             relBuilder.alias(relBuilder.field(0), "f"))
>         .project(
>             relBuilder.alias(relBuilder.field(0), "f"))
>         .build();
>     RelOptPlanner planner = relNode.getCluster().getPlanner();
>     RuleSet ruleSet =
>         RuleSets.ofList(
>             ReduceExpressionsRule.PROJECT_INSTANCE,
>             new ProjectMergeRuleWithLongerName(),
>             EnumerableRules.ENUMERABLE_PROJECT_RULE,
>             EnumerableRules.ENUMERABLE_VALUES_RULE);
>     Program program = Programs.of(ruleSet);
>     RelTraitSet toTraits =
>         relNode.getCluster().traitSet()
>             .replace(0, EnumerableConvention.INSTANCE);
>     RelNode output = program.run(planner, relNode, toTraits,
>         ImmutableList.<RelOptMaterialization>of(), ImmutableList.<RelOptLattice>of());
>     // check for output
>   }
>   /**
>    * ProjectMergeRule inheritor which has
>    * class name greater than ProjectReduceExpressionsRule class name (String.compareTo()).
>    *
>    * It is needed for RuleQueue.popMatch() method
>    * to apply this rule before ProjectReduceExpressionsRule.
>    */
>   private static class ProjectMergeRuleWithLongerName extends ProjectMergeRule {
>     public ProjectMergeRuleWithLongerName() {
>       super(true, RelFactories.LOGICAL_BUILDER);
>     }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)