You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@calcite.apache.org by Timothy Farkas <tf...@mapr.com> on 2017/08/14 19:39:25 UTC

Multiple Rules Matching LogicalProject

Hi All,

I'm new to Calcite, so this may be a silly question. How can I control the behavior of the VolcanoPlanner when two rules match the same relation node and transform it? I am seeing an issue where I have two Rules (RuleA and RuleB) which match a LogicalProject relation node. When RuleA matches it transforms the relational node to a ProjectA node, and when RuleB matches it transforms the relational node to a ProjectB node. However, the output of the planner always has ProjectA nodes and never has ProjectB nodes. How can I tell the planner than I prefer ProjectB nodes over ProjectA nodes?

I am using the Apache Drill version of Calcite which is based on Calcite 1.4

Thanks,
Tim

Re: Multiple Rules Matching LogicalProject

Posted by Timothy Farkas <tf...@mapr.com>.

Thanks for the explanation Julian. I'll give tracing a shot as well.

Thanks,
Tim

________________________________
From: Julian Hyde <jh...@apache.org>
Sent: Tuesday, August 22, 2017 9:49:53 PM
To: dev@calcite.apache.org
Subject: Re: Multiple Rules Matching LogicalProject

Tim,

What you are describing is intended behavior. The whole point of a genuine cost-based optimizer is that rather than having to choose whether to take road A or road B, it can take both routes, and see which is the shortest.

The Volcano planning algorithm does not consider cost when it has identified that it can apply a rule; it just fires the rule. Rules don’t modify relational expressions - they create new ones, perhaps sharing some of the same inputs as the old expression - so it all works out.

So, Zain is correct. You *do* want the planner to produce both ProjectA and ProjectB. You just need to adjust your cost model so that it chooses the right one. Turning on tracing[1] may help.

Julian

[1] https://calcite.apache.org/docs/howto.html#tracing <https://calcite.apache.org/docs/howto.html#tracing>

> On Aug 14, 2017, at 2:54 PM, Timothy Farkas <tf...@mapr.com> wrote:
>
>
> Thanks Zain,
>
> I'll take a look at setting the cost.
>
>
> ________________________________
> From: Zain Humayun <zh...@oath.com.INVALID>
> Sent: Monday, August 14, 2017 12:45:17 PM
> To: dev@calcite.apache.org
> Subject: Re: Multiple Rules Matching LogicalProject
>
> I'm not too sure how the planner deals with tie breakers between two plans,
> but if you prefer ProjectB to ProjectA, then giving ProjectB a lower cost
> than ProjectA should result in the planner choosing ProjectB.
>
> Zain.
>
> On Mon, Aug 14, 2017 at 12:39 PM, Timothy Farkas <tf...@mapr.com> wrote:
>
>> Hi All,
>>
>> I'm new to Calcite, so this may be a silly question. How can I control the
>> behavior of the VolcanoPlanner when two rules match the same relation node
>> and transform it? I am seeing an issue where I have two Rules (RuleA and
>> RuleB) which match a LogicalProject relation node. When RuleA matches it
>> transforms the relational node to a ProjectA node, and when RuleB matches
>> it transforms the relational node to a ProjectB node. However, the output
>> of the planner always has ProjectA nodes and never has ProjectB nodes. How
>> can I tell the planner than I prefer ProjectB nodes over ProjectA nodes?
>>
>> I am using the Apache Drill version of Calcite which is based on Calcite
>> 1.4
>>
>> Thanks,
>> Tim
>>

Re: Multiple Rules Matching LogicalProject

Posted by Julian Hyde <jh...@apache.org>.

Tim,

What you are describing is intended behavior. The whole point of a genuine cost-based optimizer is that rather than having to choose whether to take road A or road B, it can take both routes, and see which is the shortest.

The Volcano planning algorithm does not consider cost when it has identified that it can apply a rule; it just fires the rule. Rules don’t modify relational expressions - they create new ones, perhaps sharing some of the same inputs as the old expression - so it all works out.

So, Zain is correct. You *do* want the planner to produce both ProjectA and ProjectB. You just need to adjust your cost model so that it chooses the right one. Turning on tracing[1] may help.

Julian

[1] https://calcite.apache.org/docs/howto.html#tracing <https://calcite.apache.org/docs/howto.html#tracing> 

> On Aug 14, 2017, at 2:54 PM, Timothy Farkas <tf...@mapr.com> wrote:
> 
> 
> Thanks Zain,
> 
> I'll take a look at setting the cost.
> 
> 
> ________________________________
> From: Zain Humayun <zh...@oath.com.INVALID>
> Sent: Monday, August 14, 2017 12:45:17 PM
> To: dev@calcite.apache.org
> Subject: Re: Multiple Rules Matching LogicalProject
> 
> I'm not too sure how the planner deals with tie breakers between two plans,
> but if you prefer ProjectB to ProjectA, then giving ProjectB a lower cost
> than ProjectA should result in the planner choosing ProjectB.
> 
> Zain.
> 
> On Mon, Aug 14, 2017 at 12:39 PM, Timothy Farkas <tf...@mapr.com> wrote:
> 
>> Hi All,
>> 
>> I'm new to Calcite, so this may be a silly question. How can I control the
>> behavior of the VolcanoPlanner when two rules match the same relation node
>> and transform it? I am seeing an issue where I have two Rules (RuleA and
>> RuleB) which match a LogicalProject relation node. When RuleA matches it
>> transforms the relational node to a ProjectA node, and when RuleB matches
>> it transforms the relational node to a ProjectB node. However, the output
>> of the planner always has ProjectA nodes and never has ProjectB nodes. How
>> can I tell the planner than I prefer ProjectB nodes over ProjectA nodes?
>> 
>> I am using the Apache Drill version of Calcite which is based on Calcite
>> 1.4
>> 
>> Thanks,
>> Tim
>>

Re: Multiple Rules Matching LogicalProject

Posted by Timothy Farkas <tf...@mapr.com>.

Thanks Zain,

I'll take a look at setting the cost.


________________________________
From: Zain Humayun <zh...@oath.com.INVALID>
Sent: Monday, August 14, 2017 12:45:17 PM
To: dev@calcite.apache.org
Subject: Re: Multiple Rules Matching LogicalProject

I'm not too sure how the planner deals with tie breakers between two plans,
but if you prefer ProjectB to ProjectA, then giving ProjectB a lower cost
than ProjectA should result in the planner choosing ProjectB.

Zain.

On Mon, Aug 14, 2017 at 12:39 PM, Timothy Farkas <tf...@mapr.com> wrote:

> Hi All,
>
> I'm new to Calcite, so this may be a silly question. How can I control the
> behavior of the VolcanoPlanner when two rules match the same relation node
> and transform it? I am seeing an issue where I have two Rules (RuleA and
> RuleB) which match a LogicalProject relation node. When RuleA matches it
> transforms the relational node to a ProjectA node, and when RuleB matches
> it transforms the relational node to a ProjectB node. However, the output
> of the planner always has ProjectA nodes and never has ProjectB nodes. How
> can I tell the planner than I prefer ProjectB nodes over ProjectA nodes?
>
> I am using the Apache Drill version of Calcite which is based on Calcite
> 1.4
>
> Thanks,
> Tim
>

Re: Multiple Rules Matching LogicalProject

Posted by Zain Humayun <zh...@oath.com.INVALID>.

I'm not too sure how the planner deals with tie breakers between two plans,
but if you prefer ProjectB to ProjectA, then giving ProjectB a lower cost
than ProjectA should result in the planner choosing ProjectB.

Zain.

On Mon, Aug 14, 2017 at 12:39 PM, Timothy Farkas <tf...@mapr.com> wrote:

> Hi All,
>
> I'm new to Calcite, so this may be a silly question. How can I control the
> behavior of the VolcanoPlanner when two rules match the same relation node
> and transform it? I am seeing an issue where I have two Rules (RuleA and
> RuleB) which match a LogicalProject relation node. When RuleA matches it
> transforms the relational node to a ProjectA node, and when RuleB matches
> it transforms the relational node to a ProjectB node. However, the output
> of the planner always has ProjectA nodes and never has ProjectB nodes. How
> can I tell the planner than I prefer ProjectB nodes over ProjectA nodes?
>
> I am using the Apache Drill version of Calcite which is based on Calcite
> 1.4
>
> Thanks,
> Tim
>