You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Muhammad Gelbana <m....@gmail.com> on 2017/08/01 12:25:09 UTC
Re: Handling "HepRelVertex" and "RelSubset" as input
I'm facing a similar problem but with Join inputs. Is there a way to
control the type of nodes returned as inputs to a join ? I wrote a rule for
a join but it failed to match it because I expect the join inputs to be
*JdbcRel* nodes while the inputs were actually *HepRelVertex*. I'm using
Drill.
- Gelbana
On Fri, Jun 30, 2017 at 4:35 PM, Muhammad Gelbana <m....@gmail.com>
wrote:
> Thanks a lot Julian. That was very helpful.
>
> *---------------------*
> *Muhammad Gelbana*
>
> On Thu, Jun 29, 2017 at 8:04 PM, Julian Hyde <jh...@apache.org> wrote:
>
>> Consider ProjectFilterTransposeRule, a simple rule that matches a Project
>> on top of a Filter. But how to get that Project and Filter?
>>
>> Run RelOptRulesTest.testPushProjectPastFilter in the debugger, and put a
>> breakpoint in ProjectFilterTransposeRule.onMatch [1]. Look at the
>> Project “origProj” in the debugger. Its input is a HepRelVertex containing
>> a Filter. But Filter “filter” is, sure enough, a Filter.
>>
>> So, to get the filter, don’t call origProj.getInput(), instead call
>> call.rel(1). The RelOptRuleCall will have ensured that this is the correct
>> type, and removed any wrappers.
>>
>> Julian
>>
>>
>> [1] https://insight.io/github.com/apache/calcite/blob/master/cor
>> e/src/main/java/org/apache/calcite/rel/rules/ProjectFilterTr
>> ansposeRule.java?line=78 <https://insight.io/github.com
>> /apache/calcite/blob/master/core/src/main/java/org/apache/
>> calcite/rel/rules/ProjectFilterTransposeRule.java?line=78>
>> > On Jun 29, 2017, at 8:47 AM, Muhammad Gelbana <m....@gmail.com>
>> wrote:
>> >
>> > But how can I handle the unexpected variation of the input class type ?
>> > For example, please consider the following *RelOptRule* constructor
>> >
>> > public IncortaLimitRule() {
>> >> super(operand(*DrillLimitRel*.class, operand(*IncortaJdbcDrel*.clas
>> s,
>> >> operand(*JdbcRel*.class, any()))), "TestRule");
>> >> }
>> >
>> >
>> > I'm trying to be precise so I specified the RelNode (*DrillLimitRel*),
>> it's
>> > input (*IncortaJdbcDrel*) and it's input's input (*JdbcRel*, this is an
>> > interface but I understand that the planner should match relnodes if the
>> > rule's operand is a parent of examined operand, so for instance,
>> > *JdbcTableScan* should match).
>> >
>> > Is there a way to guarantee the relnode's input type ? Looking into
>> other
>> > rules, I don't remember seeing type checking for input relnodes. Is it a
>> > good sign that I have to do multiple checks for the input relnode type ?
>> >
>> > *---------------------*
>> > *Muhammad Gelbana*
>> >
>> > On Tue, Jun 27, 2017 at 1:36 AM, Julian Hyde <jh...@apache.org> wrote:
>> >
>> >> RelSubset provides a level of indirection that allows Calcite to
>> optimize
>> >> queries. If the input to a relational operator is an equivalence
>> class, not
>> >> a particular relational expression, then Calcite has the freedom to
>> choose
>> >> the member of the equivalence class that has the cheapest cost.
>> >>
>> >> Calcite uses the Volcano algorithm, a form of dynamic programming, to
>> do
>> >> this.
>> >>
>> >> As you have discovered it means that you cannot simply tree-walk over a
>> >> relational expression, looking for, say, a Project on top of a Filter.
>> You
>> >> need to write a RelOptRule that declares that it wants to see a
>> Project on
>> >> top of a Filter, and Volcano will fire that rule whenever that
>> combination
>> >> arises.
>> >>
>> >> Julian
>> >>
>> >>
>> >>> On Jun 26, 2017, at 4:00 PM, Muhammad Gelbana <m....@gmail.com>
>> >> wrote:
>> >>>
>> >>> While debugging through some of the rules I'm trying to get to work, I
>> >> find
>> >>> that the matched relnode's input is *HepRelVertex* or a *RelSubset*.
>> >>>
>> >>> Why wouldn't I be getting a descendant of the following types:
>> >>>
>> >>> - BiRel
>> >>> - MultiJoin
>> >>> - SetOp
>> >>> - SingleRel
>> >>> - TableScan
>> >>>
>> >>> There are other types too but I can assume how to handle the types I
>> >> listed.
>> >>>
>> >>> *RelSubset* is really confusing because, as far as I understand, it's
>> a
>> >>> group of relnodes. So how can I decide which node to get from this
>> group
>> >> so
>> >>> I can use it as an input to the new relnode I'm creating ?! Or should
>> I
>> >>> pass the *RelSubset* itself ? Why am I getting it in the first place
>> and
>> >>> not a single relnode ?
>> >>>
>> >>> *---------------------*
>> >>> *Muhammad Gelbana*
>> >>
>> >>
>>
>>
>
Re: Handling "HepRelVertex" and "RelSubset" as input
Posted by Julian Hyde <jh...@apache.org>.
Yes, to repeat: don’t call RelNode.getInput(int), or RelNode.getInputs(), or Join.getLeft() or Join.getRight(), or SingleRelNode.getInput(). If you do, you will get the wrapped input, a HepRelVertex or a RelSubset.
Instead call RelOptRuleCall.rel(int).
Julian
> On Aug 2, 2017, at 1:34 PM, Muhammad Gelbana <m....@gmail.com> wrote:
>
> Of course! You are correct. I got used to single input nodes and I didn't
> deal with the Join properly.
>
> I get the inputs of the join using *join.getInputs()*.
>
>
>
> Thanks,
> Gelbana
>
> On Wed, Aug 2, 2017 at 8:10 PM, Julian Hyde <jh...@apache.org> wrote:
>
>> Join has two inputs but you have only one “any()”. I am surprised that it
>> is matching a Join at all (an operand with one input should not match a
>> RelNode with two inputs). But given that it does… you have two operands, so
>> call.rel(0) and call.rel(1) will be valid but call.rel(2) will not.
>>
>> So, how are you getting the right input to the join? Are you calling
>> Join.getInput(1) or Join.getRight()? Yes, those will be HepRelVertex or
>> RelSubset, as you are seeing.
>>
>> You should add another “, any()”, then call call.rel(2) when the rule
>> matches.
>>
>> Julian
>>
>>
>>
>>> On Aug 2, 2017, at 3:15 AM, Muhammad Gelbana <m....@gmail.com>
>> wrote:
>>>
>>> Here it is
>>>
>>>> public GelbanaJoinRule(JdbcConvention out) {
>>>> super(operand(LogicalJoin.class, Convention.NONE, any()),
>>>> "GelbanaJoinRule");
>>>> this.outConvention = out;
>>>> }
>>>
>>>
>>> The nodes appearing as *HepRelVertex* are the *LogicalJoin*'s input
>> nodes,
>>> not the operands passed to the rule using the
>>> *org.apache.calcite.plan.RelOptRule.matches(RelOptRuleCall)* method.
>>>
>>>
>>>
>>> - Gelbana
>>>
>>> On Tue, Aug 1, 2017 at 6:13 PM, Jess Balint <jb...@gmail.com> wrote:
>>>
>>>> Can you share your operand constraints? Where did you see HepRelVertex?
>> As
>>>> Julian described, the rel nodes are of the correct type when matched
>> with
>>>> operand constraints.
>>>>
>>>> Jess
>>>>
>>>> On Tue, Aug 1, 2017 at 7:25 AM, Muhammad Gelbana <m....@gmail.com>
>>>> wrote:
>>>>
>>>>> I'm facing a similar problem but with Join inputs. Is there a way to
>>>>> control the type of nodes returned as inputs to a join ? I wrote a rule
>>>> for
>>>>> a join but it failed to match it because I expect the join inputs to be
>>>>> *JdbcRel* nodes while the inputs were actually *HepRelVertex*. I'm
>> using
>>>>> Drill.
>>>>>
>>>>>
>>>>>
>>>>> - Gelbana
>>>>>
>>>>> On Fri, Jun 30, 2017 at 4:35 PM, Muhammad Gelbana <m.gelbana@gmail.com
>>>
>>>>> wrote:
>>>>>
>>>>>> Thanks a lot Julian. That was very helpful.
>>>>>>
>>>>>> *---------------------*
>>>>>> *Muhammad Gelbana*
>>>>>>
>>>>>> On Thu, Jun 29, 2017 at 8:04 PM, Julian Hyde <jh...@apache.org>
>> wrote:
>>>>>>
>>>>>>> Consider ProjectFilterTransposeRule, a simple rule that matches a
>>>>> Project
>>>>>>> on top of a Filter. But how to get that Project and Filter?
>>>>>>>
>>>>>>> Run RelOptRulesTest.testPushProjectPastFilter in the debugger, and
>>>> put
>>>>> a
>>>>>>> breakpoint in ProjectFilterTransposeRule.onMatch [1]. Look at the
>>>>>>> Project “origProj” in the debugger. Its input is a HepRelVertex
>>>>> containing
>>>>>>> a Filter. But Filter “filter” is, sure enough, a Filter.
>>>>>>>
>>>>>>> So, to get the filter, don’t call origProj.getInput(), instead call
>>>>>>> call.rel(1). The RelOptRuleCall will have ensured that this is the
>>>>> correct
>>>>>>> type, and removed any wrappers.
>>>>>>>
>>>>>>> Julian
>>>>>>>
>>>>>>>
>>>>>>> [1] https://insight.io/github.com/apache/calcite/blob/master/cor
>>>>>>> e/src/main/java/org/apache/calcite/rel/rules/ProjectFilterTr
>>>>>>> ansposeRule.java?line=78 <https://insight.io/github.com
>>>>>>> /apache/calcite/blob/master/core/src/main/java/org/apache/
>>>>>>> calcite/rel/rules/ProjectFilterTransposeRule.java?line=78>
>>>>>>>> On Jun 29, 2017, at 8:47 AM, Muhammad Gelbana <m....@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> But how can I handle the unexpected variation of the input class
>>>>> type ?
>>>>>>>> For example, please consider the following *RelOptRule* constructor
>>>>>>>>
>>>>>>>> public IncortaLimitRule() {
>>>>>>>>> super(operand(*DrillLimitRel*.class,
>>>>> operand(*IncortaJdbcDrel*.clas
>>>>>>> s,
>>>>>>>>> operand(*JdbcRel*.class, any()))), "TestRule");
>>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm trying to be precise so I specified the RelNode
>>>> (*DrillLimitRel*),
>>>>>>> it's
>>>>>>>> input (*IncortaJdbcDrel*) and it's input's input (*JdbcRel*, this is
>>>>> an
>>>>>>>> interface but I understand that the planner should match relnodes if
>>>>> the
>>>>>>>> rule's operand is a parent of examined operand, so for instance,
>>>>>>>> *JdbcTableScan* should match).
>>>>>>>>
>>>>>>>> Is there a way to guarantee the relnode's input type ? Looking into
>>>>>>> other
>>>>>>>> rules, I don't remember seeing type checking for input relnodes. Is
>>>>> it a
>>>>>>>> good sign that I have to do multiple checks for the input relnode
>>>>> type ?
>>>>>>>>
>>>>>>>> *---------------------*
>>>>>>>> *Muhammad Gelbana*
>>>>>>>>
>>>>>>>> On Tue, Jun 27, 2017 at 1:36 AM, Julian Hyde <jh...@apache.org>
>>>>> wrote:
>>>>>>>>
>>>>>>>>> RelSubset provides a level of indirection that allows Calcite to
>>>>>>> optimize
>>>>>>>>> queries. If the input to a relational operator is an equivalence
>>>>>>> class, not
>>>>>>>>> a particular relational expression, then Calcite has the freedom to
>>>>>>> choose
>>>>>>>>> the member of the equivalence class that has the cheapest cost.
>>>>>>>>>
>>>>>>>>> Calcite uses the Volcano algorithm, a form of dynamic programming,
>>>> to
>>>>>>> do
>>>>>>>>> this.
>>>>>>>>>
>>>>>>>>> As you have discovered it means that you cannot simply tree-walk
>>>>> over a
>>>>>>>>> relational expression, looking for, say, a Project on top of a
>>>>> Filter.
>>>>>>> You
>>>>>>>>> need to write a RelOptRule that declares that it wants to see a
>>>>>>> Project on
>>>>>>>>> top of a Filter, and Volcano will fire that rule whenever that
>>>>>>> combination
>>>>>>>>> arises.
>>>>>>>>>
>>>>>>>>> Julian
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Jun 26, 2017, at 4:00 PM, Muhammad Gelbana <
>>>> m.gelbana@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> While debugging through some of the rules I'm trying to get to
>>>>> work, I
>>>>>>>>> find
>>>>>>>>>> that the matched relnode's input is *HepRelVertex* or a
>>>> *RelSubset*.
>>>>>>>>>>
>>>>>>>>>> Why wouldn't I be getting a descendant of the following types:
>>>>>>>>>>
>>>>>>>>>> - BiRel
>>>>>>>>>> - MultiJoin
>>>>>>>>>> - SetOp
>>>>>>>>>> - SingleRel
>>>>>>>>>> - TableScan
>>>>>>>>>>
>>>>>>>>>> There are other types too but I can assume how to handle the
>>>> types I
>>>>>>>>> listed.
>>>>>>>>>>
>>>>>>>>>> *RelSubset* is really confusing because, as far as I understand,
>>>>> it's
>>>>>>> a
>>>>>>>>>> group of relnodes. So how can I decide which node to get from this
>>>>>>> group
>>>>>>>>> so
>>>>>>>>>> I can use it as an input to the new relnode I'm creating ?! Or
>>>>> should
>>>>>>> I
>>>>>>>>>> pass the *RelSubset* itself ? Why am I getting it in the first
>>>> place
>>>>>>> and
>>>>>>>>>> not a single relnode ?
>>>>>>>>>>
>>>>>>>>>> *---------------------*
>>>>>>>>>> *Muhammad Gelbana*
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>
Re: Handling "HepRelVertex" and "RelSubset" as input
Posted by Muhammad Gelbana <m....@gmail.com>.
Of course! You are correct. I got used to single input nodes and I didn't
deal with the Join properly.
I get the inputs of the join using *join.getInputs()*.
Thanks,
Gelbana
On Wed, Aug 2, 2017 at 8:10 PM, Julian Hyde <jh...@apache.org> wrote:
> Join has two inputs but you have only one “any()”. I am surprised that it
> is matching a Join at all (an operand with one input should not match a
> RelNode with two inputs). But given that it does… you have two operands, so
> call.rel(0) and call.rel(1) will be valid but call.rel(2) will not.
>
> So, how are you getting the right input to the join? Are you calling
> Join.getInput(1) or Join.getRight()? Yes, those will be HepRelVertex or
> RelSubset, as you are seeing.
>
> You should add another “, any()”, then call call.rel(2) when the rule
> matches.
>
> Julian
>
>
>
> > On Aug 2, 2017, at 3:15 AM, Muhammad Gelbana <m....@gmail.com>
> wrote:
> >
> > Here it is
> >
> >> public GelbanaJoinRule(JdbcConvention out) {
> >> super(operand(LogicalJoin.class, Convention.NONE, any()),
> >> "GelbanaJoinRule");
> >> this.outConvention = out;
> >> }
> >
> >
> > The nodes appearing as *HepRelVertex* are the *LogicalJoin*'s input
> nodes,
> > not the operands passed to the rule using the
> > *org.apache.calcite.plan.RelOptRule.matches(RelOptRuleCall)* method.
> >
> >
> >
> > - Gelbana
> >
> > On Tue, Aug 1, 2017 at 6:13 PM, Jess Balint <jb...@gmail.com> wrote:
> >
> >> Can you share your operand constraints? Where did you see HepRelVertex?
> As
> >> Julian described, the rel nodes are of the correct type when matched
> with
> >> operand constraints.
> >>
> >> Jess
> >>
> >> On Tue, Aug 1, 2017 at 7:25 AM, Muhammad Gelbana <m....@gmail.com>
> >> wrote:
> >>
> >>> I'm facing a similar problem but with Join inputs. Is there a way to
> >>> control the type of nodes returned as inputs to a join ? I wrote a rule
> >> for
> >>> a join but it failed to match it because I expect the join inputs to be
> >>> *JdbcRel* nodes while the inputs were actually *HepRelVertex*. I'm
> using
> >>> Drill.
> >>>
> >>>
> >>>
> >>> - Gelbana
> >>>
> >>> On Fri, Jun 30, 2017 at 4:35 PM, Muhammad Gelbana <m.gelbana@gmail.com
> >
> >>> wrote:
> >>>
> >>>> Thanks a lot Julian. That was very helpful.
> >>>>
> >>>> *---------------------*
> >>>> *Muhammad Gelbana*
> >>>>
> >>>> On Thu, Jun 29, 2017 at 8:04 PM, Julian Hyde <jh...@apache.org>
> wrote:
> >>>>
> >>>>> Consider ProjectFilterTransposeRule, a simple rule that matches a
> >>> Project
> >>>>> on top of a Filter. But how to get that Project and Filter?
> >>>>>
> >>>>> Run RelOptRulesTest.testPushProjectPastFilter in the debugger, and
> >> put
> >>> a
> >>>>> breakpoint in ProjectFilterTransposeRule.onMatch [1]. Look at the
> >>>>> Project “origProj” in the debugger. Its input is a HepRelVertex
> >>> containing
> >>>>> a Filter. But Filter “filter” is, sure enough, a Filter.
> >>>>>
> >>>>> So, to get the filter, don’t call origProj.getInput(), instead call
> >>>>> call.rel(1). The RelOptRuleCall will have ensured that this is the
> >>> correct
> >>>>> type, and removed any wrappers.
> >>>>>
> >>>>> Julian
> >>>>>
> >>>>>
> >>>>> [1] https://insight.io/github.com/apache/calcite/blob/master/cor
> >>>>> e/src/main/java/org/apache/calcite/rel/rules/ProjectFilterTr
> >>>>> ansposeRule.java?line=78 <https://insight.io/github.com
> >>>>> /apache/calcite/blob/master/core/src/main/java/org/apache/
> >>>>> calcite/rel/rules/ProjectFilterTransposeRule.java?line=78>
> >>>>>> On Jun 29, 2017, at 8:47 AM, Muhammad Gelbana <m....@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> But how can I handle the unexpected variation of the input class
> >>> type ?
> >>>>>> For example, please consider the following *RelOptRule* constructor
> >>>>>>
> >>>>>> public IncortaLimitRule() {
> >>>>>>> super(operand(*DrillLimitRel*.class,
> >>> operand(*IncortaJdbcDrel*.clas
> >>>>> s,
> >>>>>>> operand(*JdbcRel*.class, any()))), "TestRule");
> >>>>>>> }
> >>>>>>
> >>>>>>
> >>>>>> I'm trying to be precise so I specified the RelNode
> >> (*DrillLimitRel*),
> >>>>> it's
> >>>>>> input (*IncortaJdbcDrel*) and it's input's input (*JdbcRel*, this is
> >>> an
> >>>>>> interface but I understand that the planner should match relnodes if
> >>> the
> >>>>>> rule's operand is a parent of examined operand, so for instance,
> >>>>>> *JdbcTableScan* should match).
> >>>>>>
> >>>>>> Is there a way to guarantee the relnode's input type ? Looking into
> >>>>> other
> >>>>>> rules, I don't remember seeing type checking for input relnodes. Is
> >>> it a
> >>>>>> good sign that I have to do multiple checks for the input relnode
> >>> type ?
> >>>>>>
> >>>>>> *---------------------*
> >>>>>> *Muhammad Gelbana*
> >>>>>>
> >>>>>> On Tue, Jun 27, 2017 at 1:36 AM, Julian Hyde <jh...@apache.org>
> >>> wrote:
> >>>>>>
> >>>>>>> RelSubset provides a level of indirection that allows Calcite to
> >>>>> optimize
> >>>>>>> queries. If the input to a relational operator is an equivalence
> >>>>> class, not
> >>>>>>> a particular relational expression, then Calcite has the freedom to
> >>>>> choose
> >>>>>>> the member of the equivalence class that has the cheapest cost.
> >>>>>>>
> >>>>>>> Calcite uses the Volcano algorithm, a form of dynamic programming,
> >> to
> >>>>> do
> >>>>>>> this.
> >>>>>>>
> >>>>>>> As you have discovered it means that you cannot simply tree-walk
> >>> over a
> >>>>>>> relational expression, looking for, say, a Project on top of a
> >>> Filter.
> >>>>> You
> >>>>>>> need to write a RelOptRule that declares that it wants to see a
> >>>>> Project on
> >>>>>>> top of a Filter, and Volcano will fire that rule whenever that
> >>>>> combination
> >>>>>>> arises.
> >>>>>>>
> >>>>>>> Julian
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Jun 26, 2017, at 4:00 PM, Muhammad Gelbana <
> >> m.gelbana@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> While debugging through some of the rules I'm trying to get to
> >>> work, I
> >>>>>>> find
> >>>>>>>> that the matched relnode's input is *HepRelVertex* or a
> >> *RelSubset*.
> >>>>>>>>
> >>>>>>>> Why wouldn't I be getting a descendant of the following types:
> >>>>>>>>
> >>>>>>>> - BiRel
> >>>>>>>> - MultiJoin
> >>>>>>>> - SetOp
> >>>>>>>> - SingleRel
> >>>>>>>> - TableScan
> >>>>>>>>
> >>>>>>>> There are other types too but I can assume how to handle the
> >> types I
> >>>>>>> listed.
> >>>>>>>>
> >>>>>>>> *RelSubset* is really confusing because, as far as I understand,
> >>> it's
> >>>>> a
> >>>>>>>> group of relnodes. So how can I decide which node to get from this
> >>>>> group
> >>>>>>> so
> >>>>>>>> I can use it as an input to the new relnode I'm creating ?! Or
> >>> should
> >>>>> I
> >>>>>>>> pass the *RelSubset* itself ? Why am I getting it in the first
> >> place
> >>>>> and
> >>>>>>>> not a single relnode ?
> >>>>>>>>
> >>>>>>>> *---------------------*
> >>>>>>>> *Muhammad Gelbana*
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: Handling "HepRelVertex" and "RelSubset" as input
Posted by Julian Hyde <jh...@apache.org>.
Join has two inputs but you have only one “any()”. I am surprised that it is matching a Join at all (an operand with one input should not match a RelNode with two inputs). But given that it does… you have two operands, so call.rel(0) and call.rel(1) will be valid but call.rel(2) will not.
So, how are you getting the right input to the join? Are you calling Join.getInput(1) or Join.getRight()? Yes, those will be HepRelVertex or RelSubset, as you are seeing.
You should add another “, any()”, then call call.rel(2) when the rule matches.
Julian
> On Aug 2, 2017, at 3:15 AM, Muhammad Gelbana <m....@gmail.com> wrote:
>
> Here it is
>
>> public GelbanaJoinRule(JdbcConvention out) {
>> super(operand(LogicalJoin.class, Convention.NONE, any()),
>> "GelbanaJoinRule");
>> this.outConvention = out;
>> }
>
>
> The nodes appearing as *HepRelVertex* are the *LogicalJoin*'s input nodes,
> not the operands passed to the rule using the
> *org.apache.calcite.plan.RelOptRule.matches(RelOptRuleCall)* method.
>
>
>
> - Gelbana
>
> On Tue, Aug 1, 2017 at 6:13 PM, Jess Balint <jb...@gmail.com> wrote:
>
>> Can you share your operand constraints? Where did you see HepRelVertex? As
>> Julian described, the rel nodes are of the correct type when matched with
>> operand constraints.
>>
>> Jess
>>
>> On Tue, Aug 1, 2017 at 7:25 AM, Muhammad Gelbana <m....@gmail.com>
>> wrote:
>>
>>> I'm facing a similar problem but with Join inputs. Is there a way to
>>> control the type of nodes returned as inputs to a join ? I wrote a rule
>> for
>>> a join but it failed to match it because I expect the join inputs to be
>>> *JdbcRel* nodes while the inputs were actually *HepRelVertex*. I'm using
>>> Drill.
>>>
>>>
>>>
>>> - Gelbana
>>>
>>> On Fri, Jun 30, 2017 at 4:35 PM, Muhammad Gelbana <m....@gmail.com>
>>> wrote:
>>>
>>>> Thanks a lot Julian. That was very helpful.
>>>>
>>>> *---------------------*
>>>> *Muhammad Gelbana*
>>>>
>>>> On Thu, Jun 29, 2017 at 8:04 PM, Julian Hyde <jh...@apache.org> wrote:
>>>>
>>>>> Consider ProjectFilterTransposeRule, a simple rule that matches a
>>> Project
>>>>> on top of a Filter. But how to get that Project and Filter?
>>>>>
>>>>> Run RelOptRulesTest.testPushProjectPastFilter in the debugger, and
>> put
>>> a
>>>>> breakpoint in ProjectFilterTransposeRule.onMatch [1]. Look at the
>>>>> Project “origProj” in the debugger. Its input is a HepRelVertex
>>> containing
>>>>> a Filter. But Filter “filter” is, sure enough, a Filter.
>>>>>
>>>>> So, to get the filter, don’t call origProj.getInput(), instead call
>>>>> call.rel(1). The RelOptRuleCall will have ensured that this is the
>>> correct
>>>>> type, and removed any wrappers.
>>>>>
>>>>> Julian
>>>>>
>>>>>
>>>>> [1] https://insight.io/github.com/apache/calcite/blob/master/cor
>>>>> e/src/main/java/org/apache/calcite/rel/rules/ProjectFilterTr
>>>>> ansposeRule.java?line=78 <https://insight.io/github.com
>>>>> /apache/calcite/blob/master/core/src/main/java/org/apache/
>>>>> calcite/rel/rules/ProjectFilterTransposeRule.java?line=78>
>>>>>> On Jun 29, 2017, at 8:47 AM, Muhammad Gelbana <m....@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> But how can I handle the unexpected variation of the input class
>>> type ?
>>>>>> For example, please consider the following *RelOptRule* constructor
>>>>>>
>>>>>> public IncortaLimitRule() {
>>>>>>> super(operand(*DrillLimitRel*.class,
>>> operand(*IncortaJdbcDrel*.clas
>>>>> s,
>>>>>>> operand(*JdbcRel*.class, any()))), "TestRule");
>>>>>>> }
>>>>>>
>>>>>>
>>>>>> I'm trying to be precise so I specified the RelNode
>> (*DrillLimitRel*),
>>>>> it's
>>>>>> input (*IncortaJdbcDrel*) and it's input's input (*JdbcRel*, this is
>>> an
>>>>>> interface but I understand that the planner should match relnodes if
>>> the
>>>>>> rule's operand is a parent of examined operand, so for instance,
>>>>>> *JdbcTableScan* should match).
>>>>>>
>>>>>> Is there a way to guarantee the relnode's input type ? Looking into
>>>>> other
>>>>>> rules, I don't remember seeing type checking for input relnodes. Is
>>> it a
>>>>>> good sign that I have to do multiple checks for the input relnode
>>> type ?
>>>>>>
>>>>>> *---------------------*
>>>>>> *Muhammad Gelbana*
>>>>>>
>>>>>> On Tue, Jun 27, 2017 at 1:36 AM, Julian Hyde <jh...@apache.org>
>>> wrote:
>>>>>>
>>>>>>> RelSubset provides a level of indirection that allows Calcite to
>>>>> optimize
>>>>>>> queries. If the input to a relational operator is an equivalence
>>>>> class, not
>>>>>>> a particular relational expression, then Calcite has the freedom to
>>>>> choose
>>>>>>> the member of the equivalence class that has the cheapest cost.
>>>>>>>
>>>>>>> Calcite uses the Volcano algorithm, a form of dynamic programming,
>> to
>>>>> do
>>>>>>> this.
>>>>>>>
>>>>>>> As you have discovered it means that you cannot simply tree-walk
>>> over a
>>>>>>> relational expression, looking for, say, a Project on top of a
>>> Filter.
>>>>> You
>>>>>>> need to write a RelOptRule that declares that it wants to see a
>>>>> Project on
>>>>>>> top of a Filter, and Volcano will fire that rule whenever that
>>>>> combination
>>>>>>> arises.
>>>>>>>
>>>>>>> Julian
>>>>>>>
>>>>>>>
>>>>>>>> On Jun 26, 2017, at 4:00 PM, Muhammad Gelbana <
>> m.gelbana@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> While debugging through some of the rules I'm trying to get to
>>> work, I
>>>>>>> find
>>>>>>>> that the matched relnode's input is *HepRelVertex* or a
>> *RelSubset*.
>>>>>>>>
>>>>>>>> Why wouldn't I be getting a descendant of the following types:
>>>>>>>>
>>>>>>>> - BiRel
>>>>>>>> - MultiJoin
>>>>>>>> - SetOp
>>>>>>>> - SingleRel
>>>>>>>> - TableScan
>>>>>>>>
>>>>>>>> There are other types too but I can assume how to handle the
>> types I
>>>>>>> listed.
>>>>>>>>
>>>>>>>> *RelSubset* is really confusing because, as far as I understand,
>>> it's
>>>>> a
>>>>>>>> group of relnodes. So how can I decide which node to get from this
>>>>> group
>>>>>>> so
>>>>>>>> I can use it as an input to the new relnode I'm creating ?! Or
>>> should
>>>>> I
>>>>>>>> pass the *RelSubset* itself ? Why am I getting it in the first
>> place
>>>>> and
>>>>>>>> not a single relnode ?
>>>>>>>>
>>>>>>>> *---------------------*
>>>>>>>> *Muhammad Gelbana*
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: Handling "HepRelVertex" and "RelSubset" as input
Posted by Muhammad Gelbana <m....@gmail.com>.
Here it is
> public GelbanaJoinRule(JdbcConvention out) {
> super(operand(LogicalJoin.class, Convention.NONE, any()),
> "GelbanaJoinRule");
> this.outConvention = out;
> }
The nodes appearing as *HepRelVertex* are the *LogicalJoin*'s input nodes,
not the operands passed to the rule using the
*org.apache.calcite.plan.RelOptRule.matches(RelOptRuleCall)* method.
- Gelbana
On Tue, Aug 1, 2017 at 6:13 PM, Jess Balint <jb...@gmail.com> wrote:
> Can you share your operand constraints? Where did you see HepRelVertex? As
> Julian described, the rel nodes are of the correct type when matched with
> operand constraints.
>
> Jess
>
> On Tue, Aug 1, 2017 at 7:25 AM, Muhammad Gelbana <m....@gmail.com>
> wrote:
>
> > I'm facing a similar problem but with Join inputs. Is there a way to
> > control the type of nodes returned as inputs to a join ? I wrote a rule
> for
> > a join but it failed to match it because I expect the join inputs to be
> > *JdbcRel* nodes while the inputs were actually *HepRelVertex*. I'm using
> > Drill.
> >
> >
> >
> > - Gelbana
> >
> > On Fri, Jun 30, 2017 at 4:35 PM, Muhammad Gelbana <m....@gmail.com>
> > wrote:
> >
> > > Thanks a lot Julian. That was very helpful.
> > >
> > > *---------------------*
> > > *Muhammad Gelbana*
> > >
> > > On Thu, Jun 29, 2017 at 8:04 PM, Julian Hyde <jh...@apache.org> wrote:
> > >
> > >> Consider ProjectFilterTransposeRule, a simple rule that matches a
> > Project
> > >> on top of a Filter. But how to get that Project and Filter?
> > >>
> > >> Run RelOptRulesTest.testPushProjectPastFilter in the debugger, and
> put
> > a
> > >> breakpoint in ProjectFilterTransposeRule.onMatch [1]. Look at the
> > >> Project “origProj” in the debugger. Its input is a HepRelVertex
> > containing
> > >> a Filter. But Filter “filter” is, sure enough, a Filter.
> > >>
> > >> So, to get the filter, don’t call origProj.getInput(), instead call
> > >> call.rel(1). The RelOptRuleCall will have ensured that this is the
> > correct
> > >> type, and removed any wrappers.
> > >>
> > >> Julian
> > >>
> > >>
> > >> [1] https://insight.io/github.com/apache/calcite/blob/master/cor
> > >> e/src/main/java/org/apache/calcite/rel/rules/ProjectFilterTr
> > >> ansposeRule.java?line=78 <https://insight.io/github.com
> > >> /apache/calcite/blob/master/core/src/main/java/org/apache/
> > >> calcite/rel/rules/ProjectFilterTransposeRule.java?line=78>
> > >> > On Jun 29, 2017, at 8:47 AM, Muhammad Gelbana <m....@gmail.com>
> > >> wrote:
> > >> >
> > >> > But how can I handle the unexpected variation of the input class
> > type ?
> > >> > For example, please consider the following *RelOptRule* constructor
> > >> >
> > >> > public IncortaLimitRule() {
> > >> >> super(operand(*DrillLimitRel*.class,
> > operand(*IncortaJdbcDrel*.clas
> > >> s,
> > >> >> operand(*JdbcRel*.class, any()))), "TestRule");
> > >> >> }
> > >> >
> > >> >
> > >> > I'm trying to be precise so I specified the RelNode
> (*DrillLimitRel*),
> > >> it's
> > >> > input (*IncortaJdbcDrel*) and it's input's input (*JdbcRel*, this is
> > an
> > >> > interface but I understand that the planner should match relnodes if
> > the
> > >> > rule's operand is a parent of examined operand, so for instance,
> > >> > *JdbcTableScan* should match).
> > >> >
> > >> > Is there a way to guarantee the relnode's input type ? Looking into
> > >> other
> > >> > rules, I don't remember seeing type checking for input relnodes. Is
> > it a
> > >> > good sign that I have to do multiple checks for the input relnode
> > type ?
> > >> >
> > >> > *---------------------*
> > >> > *Muhammad Gelbana*
> > >> >
> > >> > On Tue, Jun 27, 2017 at 1:36 AM, Julian Hyde <jh...@apache.org>
> > wrote:
> > >> >
> > >> >> RelSubset provides a level of indirection that allows Calcite to
> > >> optimize
> > >> >> queries. If the input to a relational operator is an equivalence
> > >> class, not
> > >> >> a particular relational expression, then Calcite has the freedom to
> > >> choose
> > >> >> the member of the equivalence class that has the cheapest cost.
> > >> >>
> > >> >> Calcite uses the Volcano algorithm, a form of dynamic programming,
> to
> > >> do
> > >> >> this.
> > >> >>
> > >> >> As you have discovered it means that you cannot simply tree-walk
> > over a
> > >> >> relational expression, looking for, say, a Project on top of a
> > Filter.
> > >> You
> > >> >> need to write a RelOptRule that declares that it wants to see a
> > >> Project on
> > >> >> top of a Filter, and Volcano will fire that rule whenever that
> > >> combination
> > >> >> arises.
> > >> >>
> > >> >> Julian
> > >> >>
> > >> >>
> > >> >>> On Jun 26, 2017, at 4:00 PM, Muhammad Gelbana <
> m.gelbana@gmail.com>
> > >> >> wrote:
> > >> >>>
> > >> >>> While debugging through some of the rules I'm trying to get to
> > work, I
> > >> >> find
> > >> >>> that the matched relnode's input is *HepRelVertex* or a
> *RelSubset*.
> > >> >>>
> > >> >>> Why wouldn't I be getting a descendant of the following types:
> > >> >>>
> > >> >>> - BiRel
> > >> >>> - MultiJoin
> > >> >>> - SetOp
> > >> >>> - SingleRel
> > >> >>> - TableScan
> > >> >>>
> > >> >>> There are other types too but I can assume how to handle the
> types I
> > >> >> listed.
> > >> >>>
> > >> >>> *RelSubset* is really confusing because, as far as I understand,
> > it's
> > >> a
> > >> >>> group of relnodes. So how can I decide which node to get from this
> > >> group
> > >> >> so
> > >> >>> I can use it as an input to the new relnode I'm creating ?! Or
> > should
> > >> I
> > >> >>> pass the *RelSubset* itself ? Why am I getting it in the first
> place
> > >> and
> > >> >>> not a single relnode ?
> > >> >>>
> > >> >>> *---------------------*
> > >> >>> *Muhammad Gelbana*
> > >> >>
> > >> >>
> > >>
> > >>
> > >
> >
>
Re: Handling "HepRelVertex" and "RelSubset" as input
Posted by Jess Balint <jb...@gmail.com>.
Can you share your operand constraints? Where did you see HepRelVertex? As
Julian described, the rel nodes are of the correct type when matched with
operand constraints.
Jess
On Tue, Aug 1, 2017 at 7:25 AM, Muhammad Gelbana <m....@gmail.com>
wrote:
> I'm facing a similar problem but with Join inputs. Is there a way to
> control the type of nodes returned as inputs to a join ? I wrote a rule for
> a join but it failed to match it because I expect the join inputs to be
> *JdbcRel* nodes while the inputs were actually *HepRelVertex*. I'm using
> Drill.
>
>
>
> - Gelbana
>
> On Fri, Jun 30, 2017 at 4:35 PM, Muhammad Gelbana <m....@gmail.com>
> wrote:
>
> > Thanks a lot Julian. That was very helpful.
> >
> > *---------------------*
> > *Muhammad Gelbana*
> >
> > On Thu, Jun 29, 2017 at 8:04 PM, Julian Hyde <jh...@apache.org> wrote:
> >
> >> Consider ProjectFilterTransposeRule, a simple rule that matches a
> Project
> >> on top of a Filter. But how to get that Project and Filter?
> >>
> >> Run RelOptRulesTest.testPushProjectPastFilter in the debugger, and put
> a
> >> breakpoint in ProjectFilterTransposeRule.onMatch [1]. Look at the
> >> Project “origProj” in the debugger. Its input is a HepRelVertex
> containing
> >> a Filter. But Filter “filter” is, sure enough, a Filter.
> >>
> >> So, to get the filter, don’t call origProj.getInput(), instead call
> >> call.rel(1). The RelOptRuleCall will have ensured that this is the
> correct
> >> type, and removed any wrappers.
> >>
> >> Julian
> >>
> >>
> >> [1] https://insight.io/github.com/apache/calcite/blob/master/cor
> >> e/src/main/java/org/apache/calcite/rel/rules/ProjectFilterTr
> >> ansposeRule.java?line=78 <https://insight.io/github.com
> >> /apache/calcite/blob/master/core/src/main/java/org/apache/
> >> calcite/rel/rules/ProjectFilterTransposeRule.java?line=78>
> >> > On Jun 29, 2017, at 8:47 AM, Muhammad Gelbana <m....@gmail.com>
> >> wrote:
> >> >
> >> > But how can I handle the unexpected variation of the input class
> type ?
> >> > For example, please consider the following *RelOptRule* constructor
> >> >
> >> > public IncortaLimitRule() {
> >> >> super(operand(*DrillLimitRel*.class,
> operand(*IncortaJdbcDrel*.clas
> >> s,
> >> >> operand(*JdbcRel*.class, any()))), "TestRule");
> >> >> }
> >> >
> >> >
> >> > I'm trying to be precise so I specified the RelNode (*DrillLimitRel*),
> >> it's
> >> > input (*IncortaJdbcDrel*) and it's input's input (*JdbcRel*, this is
> an
> >> > interface but I understand that the planner should match relnodes if
> the
> >> > rule's operand is a parent of examined operand, so for instance,
> >> > *JdbcTableScan* should match).
> >> >
> >> > Is there a way to guarantee the relnode's input type ? Looking into
> >> other
> >> > rules, I don't remember seeing type checking for input relnodes. Is
> it a
> >> > good sign that I have to do multiple checks for the input relnode
> type ?
> >> >
> >> > *---------------------*
> >> > *Muhammad Gelbana*
> >> >
> >> > On Tue, Jun 27, 2017 at 1:36 AM, Julian Hyde <jh...@apache.org>
> wrote:
> >> >
> >> >> RelSubset provides a level of indirection that allows Calcite to
> >> optimize
> >> >> queries. If the input to a relational operator is an equivalence
> >> class, not
> >> >> a particular relational expression, then Calcite has the freedom to
> >> choose
> >> >> the member of the equivalence class that has the cheapest cost.
> >> >>
> >> >> Calcite uses the Volcano algorithm, a form of dynamic programming, to
> >> do
> >> >> this.
> >> >>
> >> >> As you have discovered it means that you cannot simply tree-walk
> over a
> >> >> relational expression, looking for, say, a Project on top of a
> Filter.
> >> You
> >> >> need to write a RelOptRule that declares that it wants to see a
> >> Project on
> >> >> top of a Filter, and Volcano will fire that rule whenever that
> >> combination
> >> >> arises.
> >> >>
> >> >> Julian
> >> >>
> >> >>
> >> >>> On Jun 26, 2017, at 4:00 PM, Muhammad Gelbana <m....@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> While debugging through some of the rules I'm trying to get to
> work, I
> >> >> find
> >> >>> that the matched relnode's input is *HepRelVertex* or a *RelSubset*.
> >> >>>
> >> >>> Why wouldn't I be getting a descendant of the following types:
> >> >>>
> >> >>> - BiRel
> >> >>> - MultiJoin
> >> >>> - SetOp
> >> >>> - SingleRel
> >> >>> - TableScan
> >> >>>
> >> >>> There are other types too but I can assume how to handle the types I
> >> >> listed.
> >> >>>
> >> >>> *RelSubset* is really confusing because, as far as I understand,
> it's
> >> a
> >> >>> group of relnodes. So how can I decide which node to get from this
> >> group
> >> >> so
> >> >>> I can use it as an input to the new relnode I'm creating ?! Or
> should
> >> I
> >> >>> pass the *RelSubset* itself ? Why am I getting it in the first place
> >> and
> >> >>> not a single relnode ?
> >> >>>
> >> >>> *---------------------*
> >> >>> *Muhammad Gelbana*
> >> >>
> >> >>
> >>
> >>
> >
>