You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by LakeShen <sh...@gmail.com> on 2023/08/14 03:47:39 UTC
[Discussion] Make SubTreeHintPropagateShuttle hint that match the search layer can be configured
Hi community, Now I am reading the source code of calcite sql hint,and I
found in SubTreeHintPropagateShuttle class, from the root node, it will be
the most search 3 layers down child nodes, trying to propagate
the original RelNode hints, 3 are fixed. SubTreeHintPropagateShuttle is a
RelShuttle,and its purpose is that when a rule rewrited a RelNode Tree,
need to propagate the original RelNode Hints to rewrite RelNode. More
details could see RelOptUtil# propagateRelHints method. My idea is that
make SubTreeHintPropagateShuttle search child layers could let user
configurable. One way is to add the setHintSearchNum and getHintSearchNum
methods to the RelOptCluster class, and then in the
RelOptUtil#propagateRelHints method, using
originalRel.getCluster().getHintSearchNum
to get the search the number of layers, and then transfer to
SubTreeHintPropagateShuttle.This is just my idea, not the final
implementation. Code links: 1.
https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L4199C7-L4202C8
2.
https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L426C2-L435C34
Best,
LakeShen
Re: [Discussion] Make SubTreeHintPropagateShuttle hint that match the search layer can be configured
Posted by LakeShen <sh...@gmail.com>.
Hi Julian,
Thanks for your reply, I have got your point,I'm going to take a look at
other engines in the industry to see how they handle hints, and when I have
an idea, we'll talk about it. Best, LakeShen
Julian Hyde <jh...@gmail.com> 于2023年8月15日周二 02:39写道：
> I haven’t had time to review your code, but I want to point out that when
> you want things to propagate up the tree, the metadata system is often the
> best fit.
>
> Hints are tricky. They originate as comments in SQL. Those comments are
> then applied to one RelNode when the SQL is translated. Now we are talking
> about propagating hints up the tree (to the direct and indirect consumers
> of a RelNode). And we also copy hints when we apply transformation rules. I
> continue to be worried that the number of hints will grow unbounded. We
> should entertain the possibility that some of these “hints” are of a
> different kind to others - explicit hints, that are part of the RelNode, vs
> derived hints, which are propagated by the metadata system (or just become
> existing forms of metadata).
>
> Julian
>
>
> > On Aug 13, 2023, at 8:51 PM, LakeShen <sh...@gmail.com> wrote:
> >
> > Sorry, there is something wrong with the above email format.You could
> check
> > it out below
> > Hi community,
> >
> > Now I am reading the source code of calcite sql hint,and I found in
> > SubTreeHintPropagateShuttle class, from the root node, it will be the
> most
> > search 3 layers down child nodes, trying to propagate
> > the original RelNode hints, 3 are fixed.
> >
> > SubTreeHintPropagateShuttle is a RelShuttle,and its purpose is that when
> a
> > rule rewrited a RelNode Tree, need to propagate the original RelNode
> Hints
> > to rewrite RelNode. More details could see RelOptUtil# propagateRelHints
> > method.
> >
> > My idea is that make SubTreeHintPropagateShuttle search child layers
> could
> > let user configurable. One way is to add the setHintSearchNum and
> > getHintSearchNum methods to the RelOptCluster class, and then in
> > the RelOptUtil#propagateRelHints method, using
> > originalRel.getCluster().getHintSearchNum
> > to get the search the number of layers, and then transfer to
> > SubTreeHintPropagateShuttle.This is just my idea, not the final
> > implementation.
> >
> > Code links: 1.
> >
> https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L4199C7-L4202C8
> > 2.
> >
> https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L426C2-L435C34
> >
> > Best,
> > LakeShen
> >
> > LakeShen <sh...@gmail.com> 于2023年8月14日周一 11:47写道：
> >
> >> Hi community, Now I am reading the source code of calcite sql hint,and I
> >> found in SubTreeHintPropagateShuttle class, from the root node, it will
> be
> >> the most search 3 layers down child nodes, trying to propagate
> >> the original RelNode hints, 3 are fixed. SubTreeHintPropagateShuttle is
> a
> >> RelShuttle,and its purpose is that when a rule rewrited a RelNode Tree,
> >> need to propagate the original RelNode Hints to rewrite RelNode. More
> >> details could see RelOptUtil# propagateRelHints method. My idea is that
> >> make SubTreeHintPropagateShuttle search child layers could let user
> >> configurable. One way is to add the setHintSearchNum and
> getHintSearchNum
> >> methods to the RelOptCluster class, and then in the
> >> RelOptUtil#propagateRelHints method, using
> originalRel.getCluster().getHintSearchNum
> >> to get the search the number of layers, and then transfer to
> >> SubTreeHintPropagateShuttle.This is just my idea, not the final
> >> implementation. Code links: 1.
> >>
> https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L4199C7-L4202C8
> >> 2.
> >>
> https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L426C2-L435C34
> >>
> >> Best,
> >> LakeShen
> >>
>
>
Re: [Discussion] Make SubTreeHintPropagateShuttle hint that match the search layer can be configured
Posted by Julian Hyde <jh...@gmail.com>.
I haven’t had time to review your code, but I want to point out that when you want things to propagate up the tree, the metadata system is often the best fit.
Hints are tricky. They originate as comments in SQL. Those comments are then applied to one RelNode when the SQL is translated. Now we are talking about propagating hints up the tree (to the direct and indirect consumers of a RelNode). And we also copy hints when we apply transformation rules. I continue to be worried that the number of hints will grow unbounded. We should entertain the possibility that some of these “hints” are of a different kind to others - explicit hints, that are part of the RelNode, vs derived hints, which are propagated by the metadata system (or just become existing forms of metadata).
Julian
> On Aug 13, 2023, at 8:51 PM, LakeShen <sh...@gmail.com> wrote:
>
> Sorry, there is something wrong with the above email format.You could check
> it out below
> Hi community,
>
> Now I am reading the source code of calcite sql hint,and I found in
> SubTreeHintPropagateShuttle class, from the root node, it will be the most
> search 3 layers down child nodes, trying to propagate
> the original RelNode hints, 3 are fixed.
>
> SubTreeHintPropagateShuttle is a RelShuttle,and its purpose is that when a
> rule rewrited a RelNode Tree, need to propagate the original RelNode Hints
> to rewrite RelNode. More details could see RelOptUtil# propagateRelHints
> method.
>
> My idea is that make SubTreeHintPropagateShuttle search child layers could
> let user configurable. One way is to add the setHintSearchNum and
> getHintSearchNum methods to the RelOptCluster class, and then in
> the RelOptUtil#propagateRelHints method, using
> originalRel.getCluster().getHintSearchNum
> to get the search the number of layers, and then transfer to
> SubTreeHintPropagateShuttle.This is just my idea, not the final
> implementation.
>
> Code links: 1.
> https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L4199C7-L4202C8
> 2.
> https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L426C2-L435C34
>
> Best,
> LakeShen
>
> LakeShen <sh...@gmail.com> 于2023年8月14日周一 11:47写道：
>
>> Hi community, Now I am reading the source code of calcite sql hint,and I
>> found in SubTreeHintPropagateShuttle class, from the root node, it will be
>> the most search 3 layers down child nodes, trying to propagate
>> the original RelNode hints, 3 are fixed. SubTreeHintPropagateShuttle is a
>> RelShuttle,and its purpose is that when a rule rewrited a RelNode Tree,
>> need to propagate the original RelNode Hints to rewrite RelNode. More
>> details could see RelOptUtil# propagateRelHints method. My idea is that
>> make SubTreeHintPropagateShuttle search child layers could let user
>> configurable. One way is to add the setHintSearchNum and getHintSearchNum
>> methods to the RelOptCluster class, and then in the
>> RelOptUtil#propagateRelHints method, using originalRel.getCluster().getHintSearchNum
>> to get the search the number of layers, and then transfer to
>> SubTreeHintPropagateShuttle.This is just my idea, not the final
>> implementation. Code links: 1.
>> https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L4199C7-L4202C8
>> 2.
>> https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L426C2-L435C34
>>
>> Best,
>> LakeShen
>>
Re: [Discussion] Make SubTreeHintPropagateShuttle hint that match the search layer can be configured
Posted by LakeShen <sh...@gmail.com>.
Sorry, there is something wrong with the above email format.You could check
it out below
Hi community,
Now I am reading the source code of calcite sql hint,and I found in
SubTreeHintPropagateShuttle class, from the root node, it will be the most
search 3 layers down child nodes, trying to propagate
the original RelNode hints, 3 are fixed.
SubTreeHintPropagateShuttle is a RelShuttle,and its purpose is that when a
rule rewrited a RelNode Tree, need to propagate the original RelNode Hints
to rewrite RelNode. More details could see RelOptUtil# propagateRelHints
method.
My idea is that make SubTreeHintPropagateShuttle search child layers could
let user configurable. One way is to add the setHintSearchNum and
getHintSearchNum methods to the RelOptCluster class, and then in
the RelOptUtil#propagateRelHints method, using
originalRel.getCluster().getHintSearchNum
to get the search the number of layers, and then transfer to
SubTreeHintPropagateShuttle.This is just my idea, not the final
implementation.
Code links: 1.
https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L4199C7-L4202C8
2.
https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L426C2-L435C34
Best,
LakeShen
LakeShen <sh...@gmail.com> 于2023年8月14日周一 11:47写道：
> Hi community, Now I am reading the source code of calcite sql hint,and I
> found in SubTreeHintPropagateShuttle class, from the root node, it will be
> the most search 3 layers down child nodes, trying to propagate
> the original RelNode hints, 3 are fixed. SubTreeHintPropagateShuttle is a
> RelShuttle,and its purpose is that when a rule rewrited a RelNode Tree,
> need to propagate the original RelNode Hints to rewrite RelNode. More
> details could see RelOptUtil# propagateRelHints method. My idea is that
> make SubTreeHintPropagateShuttle search child layers could let user
> configurable. One way is to add the setHintSearchNum and getHintSearchNum
> methods to the RelOptCluster class, and then in the
> RelOptUtil#propagateRelHints method, using originalRel.getCluster().getHintSearchNum
> to get the search the number of layers, and then transfer to
> SubTreeHintPropagateShuttle.This is just my idea, not the final
> implementation. Code links: 1.
> https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L4199C7-L4202C8
> 2.
> https://github.com/apache/calcite/blob/50c3edfc3d6630528ab51fe836bd50df82cc7db8/core/src/main/java/org/apache/calcite/plan/RelOptUtil.java#L426C2-L435C34
>
> Best,
> LakeShen
>