[jira] [Comment Edited] (CALCITE-4501) Clear costs in VolcanoPlanner when setting the new root

["Julian Hyde (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CALCITE-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17285434#comment-17285434 ] 

Julian Hyde edited comment on CALCITE-4501 at 2/16/21, 8:12 PM:
----------------------------------------------------------------

It's "not valid" in the sense that, for some people that use the optimizer in particular ways, it is unnecessary and wasteful to recompute all of the costs.

No question, there is a valid use case here. When switching cost models (e.g. from a model with no traits - i.e. purely logical - to one where calling convention and sorting are traits) you clearly need to throw out the old costs. But sometimes you want to run one set of rules over the tree, and then run another set of rules over the same tree, and there's no need to wipe away costs.

I think it's time for a discussion of what is "multi-phase optimization" and how Calcite can support it.

Look into the idea of a "planner", its "current trait set" and "cost factory", and maybe add some APIs/constraints to make that lifecycle more explicit. Also take a look at what was removed in CALCITE-4111; it aimed to address this problem, but we weren't using it at this time.


was (Author: julianhyde):
It's "not valid" in the sense that, for some people that use the optimizer in particular ways, it is unnecessary and wasteful to recompute all of the costs.

> Clear costs in VolcanoPlanner when setting the new root
> -------------------------------------------------------
>
>                 Key: CALCITE-4501
>                 URL: https://issues.apache.org/jira/browse/CALCITE-4501
>             Project: Calcite
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 1.26.0
>            Reporter: Vladimir Ozerov
>            Assignee: Vladimir Ozerov
>            Priority: Minor
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> In production-grade systems, the optimization is typically performed in several phases. The same instance of {{VolcanoPlanner}} tends to be re-used between the phases. 
> In some sophisticated scenarios, we may want to alter the cost function between phases. 
> For example, to adjust the weights of some operators depending on some heuristics, or even switch to a completely different cost model. 
> One way to do that is to re-instantiate the cluster and the planner with the new cost function. 
> However, it requires copying of the current rel tree, because its nodes still point to the old cluster. This may negatively affect the performance. 
> The alternative approach could be to change the state of the cost factory. However, it would not work for two reasons:
> # The cluster object has a reference to {{MetadataQuery}} with possibly cached costs. This may lead to a mix of different cost models.
> # {{VolcanoPlanner}} caches zero and infinite costs in a final variable.
> In this issue, I propose to introduce two slight modifications to the {{VolcanoPlanner}}:
> # Clear the metadata query from within {{setRoot}} method.
> # Make {{infCost}} and {{zeroCost}} non-final, and also re-initialize them in the {{setRoot}} method.
> The {{setRoot}} method is convenient for these purposes because it is invoked from the {{RuleSetProgram}}, so the desired cost cleanup will work with no modifications to the application's code.
> Please let me know if you think that this change is not valid.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)