You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by rahul patwari <ra...@gmail.com> on 2019/08/09 12:48:23 UTC
Match Converter Rule based on Child Nodes
Hi,
We want to create a ConverterRule which converts the default calling
Convention to external storage-specific calling convention depending on the
Children nodes, like RelOptRule.
For example, depending on the properties of the child nodes, we want to
convert LogicalJoin to external system's specific Join implementation.
Currently, ConverterRule
<https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75>
cannot take Children and Child Policy is RelOptRuleOperandChildPolicy.ANY.
What is the preferred way to achieve this task?
Thanks,
Rahul
Re: Match Converter Rule based on Child Nodes
Posted by XING JIN <ji...@gmail.com>.
Hmm,, I get it
So maybe you can try HepPlanner with match order of BOTTOM_UP;
Or if VolcanoPlanner is necessary, how about call the optimization multiple
times, e.g. wrap it with a loop
Jin
rahul patwari <ra...@gmail.com> 于2019年8月23日周五 上午10:59写道：
> Hi Jin,
>
> We wanted to transform LogicalJoin to different Join Types in the external
> system depending on the LogicalJoin Child Properties(which are in external
> convention).
> As LogicalJoin can be a child of LogicalJoin, because of VolcanoPlanner's
> top-down approach, the child LogicalJoin is not converted first and we end
> up in "Not enough rules to find the Cheapest Plan".
>
> Regards,
> Rahul
>
> On Fri, Aug 23, 2019 at 8:17 AM XING JIN <ji...@gmail.com> wrote:
>
> > If I understand correctly, you can try below steps:
> > 1. Create a rule of Converter to convert child nodes to append external
> > Convention;
> > 2. Create a rule of RelOptRule to convert the parent node -- check the
> > Convention of child node when matching
> >
> > Is it applicable ?
> >
> > rahul patwari <ra...@gmail.com> 于2019年8月22日周四 下午11:18写道：
> >
> > > Hi,
> > >
> > > The properties of the child nodes correspond to the external
> Convention.
> > > So, the child nodes have to be converted before the parent can be
> > > transformed.
> > > If the property doesn't match (or) if the property cannot be
> > > determined(child node not converted), the rule cannot be applied. So,
> we
> > > end up in "Not enough Rules to find the Cheapest Plan".
> > > Is there a way to convert the Child nodes before the parent?
> > > Can VolcanoPlanner be configured to work in bottom-up fashion?
> > >
> > > Regards,
> > > Rahul
> > >
> > > On Thu, Aug 22, 2019 at 3:33 PM XING JIN <ji...@gmail.com>
> > wrote:
> > >
> > > > I guess in Rahul's case, the child node of join should be converted
> > > first,
> > > > thus physical properties will be appended.
> > > > But RelNode doesn't have a reference to parent, thus an independent
> > > > RelOptRule is needed to convert LogicalJoin.
> > > > Just as Michael said, you need to override the method of
> > > RelOptRule#matches
> > > > and check the properties of child node.
> > > >
> > > > Best,
> > > > Jin
> > > >
> > > > Stamatis Zampetakis <za...@gmail.com> 于2019年8月21日周三 下午5:50写道：
> > > >
> > > > > The Volcano planner works in a top-down fashion. It starts by
> > > > transforming
> > > > > the root and move towards the leafs of the plan. Due to this when
> > > > > transforming a logical join its inputs are still in the logical
> > > > convention
> > > > > so in principle they should not have any physical properties.
> > > > >
> > > > > Normally when the physical join algorithm is chosen the respective
> > rule
> > > > > should be responsible for demanding the inputs of the join to have
> > > > certain
> > > > > properties.
> > > > >
> > > > > Long story short, I think in your use case you should not make the
> > rule
> > > > > match based on the properties of the child nodes but it should
> match
> > > > > unconditionally and if necessary demand some properties in its
> > inputs.
> > > > If I
> > > > > remember well the EnumerableMergeJoinRule follows this approach so
> > you
> > > > can
> > > > > have a look there.
> > > > >
> > > > > Best,
> > > > > Stamatis
> > > > >
> > > > > On Tue, Aug 20, 2019, 5:07 PM Michael Mior <mm...@apache.org>
> wrote:
> > > > >
> > > > > > If you just want to control whether the rule gets applied, you
> can
> > > > > > override RelOptRule#matches which canreturns a boolean indicating
> > > > > > whether the rule should be applied.
> > > > > > --
> > > > > > Michael Mior
> > > > > > mmior@apache.org
> > > > > >
> > > > > > Le ven. 9 août 2019 à 08:48, rahul patwari
> > > > > > <ra...@gmail.com> a écrit :
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > We want to create a ConverterRule which converts the default
> > > calling
> > > > > > > Convention to external storage-specific calling convention
> > > depending
> > > > on
> > > > > > the
> > > > > > > Children nodes, like RelOptRule.
> > > > > > >
> > > > > > > For example, depending on the properties of the child nodes, we
> > > want
> > > > to
> > > > > > > convert LogicalJoin to external system's specific Join
> > > > implementation.
> > > > > > >
> > > > > > > Currently, ConverterRule
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > > > > > >
> > > > > > > cannot take Children and Child Policy is
> > > > > > RelOptRuleOperandChildPolicy.ANY.
> > > > > > >
> > > > > > > What is the preferred way to achieve this task?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Rahul
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Match Converter Rule based on Child Nodes
Posted by rahul patwari <ra...@gmail.com>.
Hi Jin,
We wanted to transform LogicalJoin to different Join Types in the external
system depending on the LogicalJoin Child Properties(which are in external
convention).
As LogicalJoin can be a child of LogicalJoin, because of VolcanoPlanner's
top-down approach, the child LogicalJoin is not converted first and we end
up in "Not enough rules to find the Cheapest Plan".
Regards,
Rahul
On Fri, Aug 23, 2019 at 8:17 AM XING JIN <ji...@gmail.com> wrote:
> If I understand correctly, you can try below steps:
> 1. Create a rule of Converter to convert child nodes to append external
> Convention;
> 2. Create a rule of RelOptRule to convert the parent node -- check the
> Convention of child node when matching
>
> Is it applicable ?
>
> rahul patwari <ra...@gmail.com> 于2019年8月22日周四 下午11:18写道：
>
> > Hi,
> >
> > The properties of the child nodes correspond to the external Convention.
> > So, the child nodes have to be converted before the parent can be
> > transformed.
> > If the property doesn't match (or) if the property cannot be
> > determined(child node not converted), the rule cannot be applied. So, we
> > end up in "Not enough Rules to find the Cheapest Plan".
> > Is there a way to convert the Child nodes before the parent?
> > Can VolcanoPlanner be configured to work in bottom-up fashion?
> >
> > Regards,
> > Rahul
> >
> > On Thu, Aug 22, 2019 at 3:33 PM XING JIN <ji...@gmail.com>
> wrote:
> >
> > > I guess in Rahul's case, the child node of join should be converted
> > first,
> > > thus physical properties will be appended.
> > > But RelNode doesn't have a reference to parent, thus an independent
> > > RelOptRule is needed to convert LogicalJoin.
> > > Just as Michael said, you need to override the method of
> > RelOptRule#matches
> > > and check the properties of child node.
> > >
> > > Best,
> > > Jin
> > >
> > > Stamatis Zampetakis <za...@gmail.com> 于2019年8月21日周三 下午5:50写道：
> > >
> > > > The Volcano planner works in a top-down fashion. It starts by
> > > transforming
> > > > the root and move towards the leafs of the plan. Due to this when
> > > > transforming a logical join its inputs are still in the logical
> > > convention
> > > > so in principle they should not have any physical properties.
> > > >
> > > > Normally when the physical join algorithm is chosen the respective
> rule
> > > > should be responsible for demanding the inputs of the join to have
> > > certain
> > > > properties.
> > > >
> > > > Long story short, I think in your use case you should not make the
> rule
> > > > match based on the properties of the child nodes but it should match
> > > > unconditionally and if necessary demand some properties in its
> inputs.
> > > If I
> > > > remember well the EnumerableMergeJoinRule follows this approach so
> you
> > > can
> > > > have a look there.
> > > >
> > > > Best,
> > > > Stamatis
> > > >
> > > > On Tue, Aug 20, 2019, 5:07 PM Michael Mior <mm...@apache.org> wrote:
> > > >
> > > > > If you just want to control whether the rule gets applied, you can
> > > > > override RelOptRule#matches which canreturns a boolean indicating
> > > > > whether the rule should be applied.
> > > > > --
> > > > > Michael Mior
> > > > > mmior@apache.org
> > > > >
> > > > > Le ven. 9 août 2019 à 08:48, rahul patwari
> > > > > <ra...@gmail.com> a écrit :
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We want to create a ConverterRule which converts the default
> > calling
> > > > > > Convention to external storage-specific calling convention
> > depending
> > > on
> > > > > the
> > > > > > Children nodes, like RelOptRule.
> > > > > >
> > > > > > For example, depending on the properties of the child nodes, we
> > want
> > > to
> > > > > > convert LogicalJoin to external system's specific Join
> > > implementation.
> > > > > >
> > > > > > Currently, ConverterRule
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > > > > >
> > > > > > cannot take Children and Child Policy is
> > > > > RelOptRuleOperandChildPolicy.ANY.
> > > > > >
> > > > > > What is the preferred way to achieve this task?
> > > > > >
> > > > > > Thanks,
> > > > > > Rahul
> > > > >
> > > >
> > >
> >
>
Re: Match Converter Rule based on Child Nodes
Posted by XING JIN <ji...@gmail.com>.
If I understand correctly, you can try below steps:
1. Create a rule of Converter to convert child nodes to append external
Convention;
2. Create a rule of RelOptRule to convert the parent node -- check the
Convention of child node when matching
Is it applicable ?
rahul patwari <ra...@gmail.com> 于2019年8月22日周四 下午11:18写道：
> Hi,
>
> The properties of the child nodes correspond to the external Convention.
> So, the child nodes have to be converted before the parent can be
> transformed.
> If the property doesn't match (or) if the property cannot be
> determined(child node not converted), the rule cannot be applied. So, we
> end up in "Not enough Rules to find the Cheapest Plan".
> Is there a way to convert the Child nodes before the parent?
> Can VolcanoPlanner be configured to work in bottom-up fashion?
>
> Regards,
> Rahul
>
> On Thu, Aug 22, 2019 at 3:33 PM XING JIN <ji...@gmail.com> wrote:
>
> > I guess in Rahul's case, the child node of join should be converted
> first,
> > thus physical properties will be appended.
> > But RelNode doesn't have a reference to parent, thus an independent
> > RelOptRule is needed to convert LogicalJoin.
> > Just as Michael said, you need to override the method of
> RelOptRule#matches
> > and check the properties of child node.
> >
> > Best,
> > Jin
> >
> > Stamatis Zampetakis <za...@gmail.com> 于2019年8月21日周三 下午5:50写道：
> >
> > > The Volcano planner works in a top-down fashion. It starts by
> > transforming
> > > the root and move towards the leafs of the plan. Due to this when
> > > transforming a logical join its inputs are still in the logical
> > convention
> > > so in principle they should not have any physical properties.
> > >
> > > Normally when the physical join algorithm is chosen the respective rule
> > > should be responsible for demanding the inputs of the join to have
> > certain
> > > properties.
> > >
> > > Long story short, I think in your use case you should not make the rule
> > > match based on the properties of the child nodes but it should match
> > > unconditionally and if necessary demand some properties in its inputs.
> > If I
> > > remember well the EnumerableMergeJoinRule follows this approach so you
> > can
> > > have a look there.
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Tue, Aug 20, 2019, 5:07 PM Michael Mior <mm...@apache.org> wrote:
> > >
> > > > If you just want to control whether the rule gets applied, you can
> > > > override RelOptRule#matches which canreturns a boolean indicating
> > > > whether the rule should be applied.
> > > > --
> > > > Michael Mior
> > > > mmior@apache.org
> > > >
> > > > Le ven. 9 août 2019 à 08:48, rahul patwari
> > > > <ra...@gmail.com> a écrit :
> > > > >
> > > > > Hi,
> > > > >
> > > > > We want to create a ConverterRule which converts the default
> calling
> > > > > Convention to external storage-specific calling convention
> depending
> > on
> > > > the
> > > > > Children nodes, like RelOptRule.
> > > > >
> > > > > For example, depending on the properties of the child nodes, we
> want
> > to
> > > > > convert LogicalJoin to external system's specific Join
> > implementation.
> > > > >
> > > > > Currently, ConverterRule
> > > > > <
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > > > >
> > > > > cannot take Children and Child Policy is
> > > > RelOptRuleOperandChildPolicy.ANY.
> > > > >
> > > > > What is the preferred way to achieve this task?
> > > > >
> > > > > Thanks,
> > > > > Rahul
> > > >
> > >
> >
>
Re: Match Converter Rule based on Child Nodes
Posted by rahul patwari <ra...@gmail.com>.
Hi,
The properties of the child nodes correspond to the external Convention.
So, the child nodes have to be converted before the parent can be
transformed.
If the property doesn't match (or) if the property cannot be
determined(child node not converted), the rule cannot be applied. So, we
end up in "Not enough Rules to find the Cheapest Plan".
Is there a way to convert the Child nodes before the parent?
Can VolcanoPlanner be configured to work in bottom-up fashion?
Regards,
Rahul
On Thu, Aug 22, 2019 at 3:33 PM XING JIN <ji...@gmail.com> wrote:
> I guess in Rahul's case, the child node of join should be converted first,
> thus physical properties will be appended.
> But RelNode doesn't have a reference to parent, thus an independent
> RelOptRule is needed to convert LogicalJoin.
> Just as Michael said, you need to override the method of RelOptRule#matches
> and check the properties of child node.
>
> Best,
> Jin
>
> Stamatis Zampetakis <za...@gmail.com> 于2019年8月21日周三 下午5:50写道：
>
> > The Volcano planner works in a top-down fashion. It starts by
> transforming
> > the root and move towards the leafs of the plan. Due to this when
> > transforming a logical join its inputs are still in the logical
> convention
> > so in principle they should not have any physical properties.
> >
> > Normally when the physical join algorithm is chosen the respective rule
> > should be responsible for demanding the inputs of the join to have
> certain
> > properties.
> >
> > Long story short, I think in your use case you should not make the rule
> > match based on the properties of the child nodes but it should match
> > unconditionally and if necessary demand some properties in its inputs.
> If I
> > remember well the EnumerableMergeJoinRule follows this approach so you
> can
> > have a look there.
> >
> > Best,
> > Stamatis
> >
> > On Tue, Aug 20, 2019, 5:07 PM Michael Mior <mm...@apache.org> wrote:
> >
> > > If you just want to control whether the rule gets applied, you can
> > > override RelOptRule#matches which canreturns a boolean indicating
> > > whether the rule should be applied.
> > > --
> > > Michael Mior
> > > mmior@apache.org
> > >
> > > Le ven. 9 août 2019 à 08:48, rahul patwari
> > > <ra...@gmail.com> a écrit :
> > > >
> > > > Hi,
> > > >
> > > > We want to create a ConverterRule which converts the default calling
> > > > Convention to external storage-specific calling convention depending
> on
> > > the
> > > > Children nodes, like RelOptRule.
> > > >
> > > > For example, depending on the properties of the child nodes, we want
> to
> > > > convert LogicalJoin to external system's specific Join
> implementation.
> > > >
> > > > Currently, ConverterRule
> > > > <
> > >
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > > >
> > > > cannot take Children and Child Policy is
> > > RelOptRuleOperandChildPolicy.ANY.
> > > >
> > > > What is the preferred way to achieve this task?
> > > >
> > > > Thanks,
> > > > Rahul
> > >
> >
>
Re: Match Converter Rule based on Child Nodes
Posted by XING JIN <ji...@gmail.com>.
I guess in Rahul's case, the child node of join should be converted first,
thus physical properties will be appended.
But RelNode doesn't have a reference to parent, thus an independent
RelOptRule is needed to convert LogicalJoin.
Just as Michael said, you need to override the method of RelOptRule#matches
and check the properties of child node.
Best,
Jin
Stamatis Zampetakis <za...@gmail.com> 于2019年8月21日周三 下午5:50写道：
> The Volcano planner works in a top-down fashion. It starts by transforming
> the root and move towards the leafs of the plan. Due to this when
> transforming a logical join its inputs are still in the logical convention
> so in principle they should not have any physical properties.
>
> Normally when the physical join algorithm is chosen the respective rule
> should be responsible for demanding the inputs of the join to have certain
> properties.
>
> Long story short, I think in your use case you should not make the rule
> match based on the properties of the child nodes but it should match
> unconditionally and if necessary demand some properties in its inputs. If I
> remember well the EnumerableMergeJoinRule follows this approach so you can
> have a look there.
>
> Best,
> Stamatis
>
> On Tue, Aug 20, 2019, 5:07 PM Michael Mior <mm...@apache.org> wrote:
>
> > If you just want to control whether the rule gets applied, you can
> > override RelOptRule#matches which canreturns a boolean indicating
> > whether the rule should be applied.
> > --
> > Michael Mior
> > mmior@apache.org
> >
> > Le ven. 9 août 2019 à 08:48, rahul patwari
> > <ra...@gmail.com> a écrit :
> > >
> > > Hi,
> > >
> > > We want to create a ConverterRule which converts the default calling
> > > Convention to external storage-specific calling convention depending on
> > the
> > > Children nodes, like RelOptRule.
> > >
> > > For example, depending on the properties of the child nodes, we want to
> > > convert LogicalJoin to external system's specific Join implementation.
> > >
> > > Currently, ConverterRule
> > > <
> >
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> > >
> > > cannot take Children and Child Policy is
> > RelOptRuleOperandChildPolicy.ANY.
> > >
> > > What is the preferred way to achieve this task?
> > >
> > > Thanks,
> > > Rahul
> >
>
Re: Match Converter Rule based on Child Nodes
Posted by Stamatis Zampetakis <za...@gmail.com>.
The Volcano planner works in a top-down fashion. It starts by transforming
the root and move towards the leafs of the plan. Due to this when
transforming a logical join its inputs are still in the logical convention
so in principle they should not have any physical properties.
Normally when the physical join algorithm is chosen the respective rule
should be responsible for demanding the inputs of the join to have certain
properties.
Long story short, I think in your use case you should not make the rule
match based on the properties of the child nodes but it should match
unconditionally and if necessary demand some properties in its inputs. If I
remember well the EnumerableMergeJoinRule follows this approach so you can
have a look there.
Best,
Stamatis
On Tue, Aug 20, 2019, 5:07 PM Michael Mior <mm...@apache.org> wrote:
> If you just want to control whether the rule gets applied, you can
> override RelOptRule#matches which canreturns a boolean indicating
> whether the rule should be applied.
> --
> Michael Mior
> mmior@apache.org
>
> Le ven. 9 août 2019 à 08:48, rahul patwari
> <ra...@gmail.com> a écrit :
> >
> > Hi,
> >
> > We want to create a ConverterRule which converts the default calling
> > Convention to external storage-specific calling convention depending on
> the
> > Children nodes, like RelOptRule.
> >
> > For example, depending on the properties of the child nodes, we want to
> > convert LogicalJoin to external system's specific Join implementation.
> >
> > Currently, ConverterRule
> > <
> https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75
> >
> > cannot take Children and Child Policy is
> RelOptRuleOperandChildPolicy.ANY.
> >
> > What is the preferred way to achieve this task?
> >
> > Thanks,
> > Rahul
>
Re: Match Converter Rule based on Child Nodes
Posted by Michael Mior <mm...@apache.org>.
If you just want to control whether the rule gets applied, you can
override RelOptRule#matches which canreturns a boolean indicating
whether the rule should be applied.
--
Michael Mior
mmior@apache.org
Le ven. 9 août 2019 à 08:48, rahul patwari
<ra...@gmail.com> a écrit :
>
> Hi,
>
> We want to create a ConverterRule which converts the default calling
> Convention to external storage-specific calling convention depending on the
> Children nodes, like RelOptRule.
>
> For example, depending on the properties of the child nodes, we want to
> convert LogicalJoin to external system's specific Join implementation.
>
> Currently, ConverterRule
> <https://github.com/apache/calcite/blob/5212d6c47e36995943f4d955a1714bf03eb08e7e/core/src/main/java/org/apache/calcite/rel/convert/ConverterRule.java#L75>
> cannot take Children and Child Policy is RelOptRuleOperandChildPolicy.ANY.
>
> What is the preferred way to achieve this task?
>
> Thanks,
> Rahul