[jira] [Commented] (CALCITE-2356) Allow user defined policy to dynamically define all or some specific rules' execution order or even skip some rules

["Vladimir Sitnikov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CALCITE-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013055#comment-17013055 ] 

Vladimir Sitnikov commented on CALCITE-2356:
--------------------------------------------

Here's a detailed step-by-step how CALCITE-2223 breaks: https://issues.apache.org/jira/browse/CALCITE-2223?focusedCommentId=16735127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16735127

TL;DR: it produces lots of relations of the following kind:
{noformat}LogicalUnion(all=[true])
  LogicalFilter(subset=[rel#363:Subset#23.NONE.[]], condition=[>=(30, $0)])
    LogicalFilter(subset=[rel#352:Subset#21.NONE.[]], condition=[>=(30, $0)])
      LogicalFilter(subset=[rel#317:Subset#19.NONE.[]], condition=[>($0, 10)])
        LogicalProject(subset=[rel#312:Subset#18.NONE.[]], deptno=[$0])
          LogicalFilter(subset=[rel#310:Subset#17.NONE.[]], condition=[>=(30, $0)])
            LogicalJoin(subset=[rel#216:Subset#4.NONE.[]], condition=[=($1, $2)],
{noformat}

In practice, it fails to recognize that extra filter adds nothing.
FilterMergeRule is there, however it is not selected for execution due to RuleMatchImportanceComparator which compares rules by Class names or something like that.

What if we classify the rules as "the rule that expands planning space" vs "the rule that shrinks planning space"?
Then we could prefer picking the rules that shrink planning space.
The classification can be either manual (e.g. a method override in the rule definition) or dynamic. For instance, the planner could count the number of new relations that was produced by a rule, then it could prefer selecting the rules that shrink the planning space (e.g. result in merges of subsets).

By expanding the planning space I mean the number of generated subsets and the number of newly added relations to the subsets.

WDYT?


> Allow user defined policy to dynamically define all or some specific rules' execution order or even skip some rules
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: CALCITE-2356
>                 URL: https://issues.apache.org/jira/browse/CALCITE-2356
>             Project: Calcite
>          Issue Type: Bug
>            Reporter: Chunhui Shi
>            Priority: Major
>
> We have seen the order of the rule execution did impact VolcanoPlanner's behavior, for example, in CALCITE-2223, if we reverse the order decided by name in RuleMatchImportanceComparator, we could get different result for CALCITE-2223 case.
> And in some of our practices, we have seen rules on overlapped patterns could also trigger unnecessary chaos and much bigger exploration space which caused the planning time became much longer.
> So the proposal of this Jira is, while each rule focuses on the local pattern, Calcite allow a pluggable coordinator of rule execution to utilize the knowledge we have about the rules and current state. The output of this coordinator is the sequence of rules to execute on matching patterns. The input is still the matching rules and pattern discovered by Calcite. When new nodes added and new rules need to be triggered, they should be added through the coordinator to decide whether/how they will be executed.
> This proposed feature should not impact any current Calcite users since they don't define their own policies.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)