You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Nikolay Vasilishin <Ni...@epam.com> on 2016/11/28 12:52:55 UTC
Convert RexSubQuery to RexNode
Hi, devs.
I'm trying to implement RexVisitor#visitSubquery method for Flink Table API. Can you help me with a problem?
1. I have to construct a RexNode while having a RexNode with left side of IN (name of column) and RelNode, which stands for table or subquery on the right side of IN operator. If I'm using RexSubQuery.in(rel, rex), I'm failing with exception:
java.lang.ClassCastException: org.apache.calcite.rex.RexSubQuery cannot be cast to org.apache.calcite.rex.RexLocalRef
at org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuilder.java:291)
at org.apache.calcite.rex.RexProgramBuilder.addCondition(RexProgramBuilder.java:265)
at org.apache.calcite.rel.rules.FilterToCalcRule.onMatch(FilterToCalcRule.java:66)
at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:213)
at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:819)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:334)
at org.apache.flink.api.table.BatchTableEnvironment.optimize(BatchTableEnvironment.scala:252)
at org.apache.flink.api.table.BatchTableEnvironment.translate(BatchTableEnvironment.scala:287)
at org.apache.flink.api.scala.table.BatchTableEnvironment.toDataSet(BatchTableEnvironment.scala:139)
at org.apache.flink.api.scala.table.TableConversions.toDataSet(TableConversions.scala:41)
at org.apache.flink.api.scala.batch.table.InITCase.testInSubquery(InITCase.scala:72)
RelBuilder#call doesn't take RelNode as parameter. Combining RelBuilder#call with output of RexSubQuery.in(rel, rex) as RexNode causes duplicated IN operator in logical plan:
LogicalFilter(condition=[IN(IN($2, {
LogicalProject(c=[$2])
LogicalFilter(condition=[=($1, 6)])
LogicalTableScan(table=[[_DataSetTable_0]])
}))])
LogicalTableScan(table=[[_DataSetTable_0]])
Besides, these classes look very similar.
So, the question is how to construct proper IN RexNode or how to cast from RexSubQuery to RexNode.
Thanks in advance,
Nikolay Vasilishin
Junior Software Engineer
Office: +7 812 611 10 94<tel:+7%20812%20611%2010%2094> x 61997<tel:61997> Cell: +7 921 403 78 55<tel:+7%20921%20403%2078%2055> Email: nikolay_vasilishin@epam.com<ma...@epam.com>
Saint-Petersburg, Russia (GMT+3) epam.com<http://www.epam.com>
CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
RE: Convert RexSubQuery to RexNode
Posted by Nikolay Vasilishin <Ni...@epam.com>.
Thanks for your help, Julian, but I think it's not exactly what I want. I have to use RexNodes while implementing logical plan in Flink engine. I found out that I have to implement Rex- and RelShuttle to convert RelNode to correct RexNode. Since it seems to be more complicated than I thought, I have to try to solve my issue in another way.
-----Original Message-----
From: Julian Hyde [mailto:jhyde@apache.org]
Sent: Monday, November 28, 2016 7:45 PM
To: dev@calcite.apache.org
Subject: Re: Convert RexSubQuery to RexNode
RelBuilder.call deals with a scalar function, and I think what you need is a table function. (In java both are just functions, but in SQL, or relational algebra, they are different beasts.)
Calcite has a TableFunctionScan, which is a RelNode, takes a function, 0 or more RelNode inputs, and 0 or more other arguments.
RelBuilder does not support TableFunctionScan yet. Can you please log a JIRA case for that? (And implement it, if you are so inclined.)
You don’t need to wait for support to be added to RelBuilder. You can create a TableFunctionScan manually, and send it to RelBuilder.push().
Julian
> On Nov 28, 2016, at 4:52 AM, Nikolay Vasilishin <Ni...@epam.com> wrote:
>
> Hi, devs.
> I'm trying to implement RexVisitor#visitSubquery method for Flink Table API. Can you help me with a problem?
>
>
> 1. I have to construct a RexNode while having a RexNode with left side of IN (name of column) and RelNode, which stands for table or subquery on the right side of IN operator. If I'm using RexSubQuery.in(rel, rex), I'm failing with exception:
>
>
>
> java.lang.ClassCastException: org.apache.calcite.rex.RexSubQuery
> cannot be cast to org.apache.calcite.rex.RexLocalRef
>
>
>
> at
> org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuild
> er.java:291)
>
> at
> org.apache.calcite.rex.RexProgramBuilder.addCondition(RexProgramBuilde
> r.java:265)
>
> at
> org.apache.calcite.rel.rules.FilterToCalcRule.onMatch(FilterToCalcRule
> .java:66)
>
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCal
> l.java:213)
>
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlan
> ner.java:819)
>
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:334
> )
>
> at
> org.apache.flink.api.table.BatchTableEnvironment.optimize(BatchTableEn
> vironment.scala:252)
>
> at
> org.apache.flink.api.table.BatchTableEnvironment.translate(BatchTableE
> nvironment.scala:287)
>
> at
> org.apache.flink.api.scala.table.BatchTableEnvironment.toDataSet(Batch
> TableEnvironment.scala:139)
>
> at
> org.apache.flink.api.scala.table.TableConversions.toDataSet(TableConve
> rsions.scala:41)
>
> at
> org.apache.flink.api.scala.batch.table.InITCase.testInSubquery(InITCas
> e.scala:72)
>
>
>
> RelBuilder#call doesn't take RelNode as parameter. Combining RelBuilder#call with output of RexSubQuery.in(rel, rex) as RexNode causes duplicated IN operator in logical plan:
>
>
>
> LogicalFilter(condition=[IN(IN($2, {
>
> LogicalProject(c=[$2])
>
> LogicalFilter(condition=[=($1, 6)])
>
> LogicalTableScan(table=[[_DataSetTable_0]])
>
> }))])
>
> LogicalTableScan(table=[[_DataSetTable_0]])
>
>
>
> Besides, these classes look very similar.
> So, the question is how to construct proper IN RexNode or how to cast from RexSubQuery to RexNode.
>
> Thanks in advance,
>
> Nikolay Vasilishin
> Junior Software Engineer
>
> Office: +7 812 611 10 94<tel:+7%20812%20611%2010%2094> x 61997<tel:61997> Cell: +7 921 403 78 55<tel:+7%20921%20403%2078%2055> Email: nikolay_vasilishin@epam.com<ma...@epam.com>
> Saint-Petersburg, Russia (GMT+3) epam.com<http://www.epam.com>
>
> CONFIDENTIALITY CAUTION AND DISCLAIMER This message is intended only
> for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
>
Re: Convert RexSubQuery to RexNode
Posted by Julian Hyde <jh...@apache.org>.
Many Thanks for logging the JIRA case. It really helps to channel discussions, when several people are running into the same problem (maybe over several months) but most people on this list are not interested, and have inboxes which are too full.
> On Nov 30, 2016, at 1:02 AM, Anton Mushin <An...@epam.com> wrote:
>
> Julian thanks for your reply!
> I created issue about it, if you have any comments about this question, please write them in issue CALCITE-1515.
>
> Best regards,
> Anton Mushin
>
> -----Original Message-----
> From: Julian Hyde [mailto:jhyde@apache.org]
> Sent: Wednesday, November 30, 2016 12:31 PM
> To: dev@calcite.apache.org
> Subject: Re: Convert RexSubQuery to RexNode
>
> Adding a TableFunctionScanFactory would be part of it. But also adding a functionScan method, analogous to the scan method but for table functions.
>
> On Wed, Nov 30, 2016 at 12:16 AM, Anton Mushin <An...@epam.com> wrote:
>> Hi Julian,
>>> RelBuilder does not support TableFunctionScan yet. Can you please log
>>> a JIRA case for that? (And implement it, if you are so inclined.)
>>
>> Do you mean add In RelBuilder TableFunctionScanFactory which will create LogicalTableFunctionScan like as for LogicalTableScan [1] and factory for others RelNodes?
>>
>> [1]
>> https://github.com/apache/calcite/blob/master/core/src/main/java/org/a
>> pache/calcite/tools/RelBuilder.java#L125
>>
>> -----Original Message-----
>> From: Julian Hyde [mailto:jhyde@apache.org]
>> Sent: Monday, November 28, 2016 8:45 PM
>> To: dev@calcite.apache.org
>> Subject: Re: Convert RexSubQuery to RexNode
>>
>> RelBuilder.call deals with a scalar function, and I think what you
>> need is a table function. (In java both are just functions, but in
>> SQL, or relational algebra, they are different beasts.)
>>
>> Calcite has a TableFunctionScan, which is a RelNode, takes a function, 0 or more RelNode inputs, and 0 or more other arguments.
>>
>> RelBuilder does not support TableFunctionScan yet. Can you please log
>> a JIRA case for that? (And implement it, if you are so inclined.)
>>
>> You don’t need to wait for support to be added to RelBuilder. You can create a TableFunctionScan manually, and send it to RelBuilder.push().
>>
>> Julian
>>
>>
>>> On Nov 28, 2016, at 4:52 AM, Nikolay Vasilishin <Ni...@epam.com> wrote:
>>>
>>> Hi, devs.
>>> I'm trying to implement RexVisitor#visitSubquery method for Flink Table API. Can you help me with a problem?
>>>
>>>
>>> 1. I have to construct a RexNode while having a RexNode with left side of IN (name of column) and RelNode, which stands for table or subquery on the right side of IN operator. If I'm using RexSubQuery.in(rel, rex), I'm failing with exception:
>>>
>>>
>>>
>>> java.lang.ClassCastException: org.apache.calcite.rex.RexSubQuery
>>> cannot be cast to org.apache.calcite.rex.RexLocalRef
>>>
>>>
>>>
>>> at
>>> org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuil
>>> d
>>> er.java:291)
>>>
>>> at
>>> org.apache.calcite.rex.RexProgramBuilder.addCondition(RexProgramBuild
>>> e
>>> r.java:265)
>>>
>>> at
>>> org.apache.calcite.rel.rules.FilterToCalcRule.onMatch(FilterToCalcRul
>>> e
>>> .java:66)
>>>
>>> at
>>> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCa
>>> l
>>> l.java:213)
>>>
>>> at
>>> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPla
>>> n
>>> ner.java:819)
>>>
>>> at
>>> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:33
>>> 4
>>> )
>>>
>>> at
>>> org.apache.flink.api.table.BatchTableEnvironment.optimize(BatchTableE
>>> n
>>> vironment.scala:252)
>>>
>>> at
>>> org.apache.flink.api.table.BatchTableEnvironment.translate(BatchTable
>>> E
>>> nvironment.scala:287)
>>>
>>> at
>>> org.apache.flink.api.scala.table.BatchTableEnvironment.toDataSet(Batc
>>> h
>>> TableEnvironment.scala:139)
>>>
>>> at
>>> org.apache.flink.api.scala.table.TableConversions.toDataSet(TableConv
>>> e
>>> rsions.scala:41)
>>>
>>> at
>>> org.apache.flink.api.scala.batch.table.InITCase.testInSubquery(InITCa
>>> s
>>> e.scala:72)
>>>
>>>
>>>
>>> RelBuilder#call doesn't take RelNode as parameter. Combining RelBuilder#call with output of RexSubQuery.in(rel, rex) as RexNode causes duplicated IN operator in logical plan:
>>>
>>>
>>>
>>> LogicalFilter(condition=[IN(IN($2, {
>>>
>>> LogicalProject(c=[$2])
>>>
>>> LogicalFilter(condition=[=($1, 6)])
>>>
>>> LogicalTableScan(table=[[_DataSetTable_0]])
>>>
>>> }))])
>>>
>>> LogicalTableScan(table=[[_DataSetTable_0]])
>>>
>>>
>>>
>>> Besides, these classes look very similar.
>>> So, the question is how to construct proper IN RexNode or how to cast from RexSubQuery to RexNode.
>>>
>>> Thanks in advance,
>>>
>>> Nikolay Vasilishin
>>> Junior Software Engineer
>>>
>>> Office: +7 812 611 10 94<tel:+7%20812%20611%2010%2094> x 61997<tel:61997> Cell: +7 921 403 78 55<tel:+7%20921%20403%2078%2055> Email: nikolay_vasilishin@epam.com<ma...@epam.com>
>>> Saint-Petersburg, Russia (GMT+3) epam.com<http://www.epam.com>
>>>
>>> CONFIDENTIALITY CAUTION AND DISCLAIMER This message is intended only
>>> for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
>>>
>>
RE: Convert RexSubQuery to RexNode
Posted by Anton Mushin <An...@epam.com>.
Julian thanks for your reply!
I created issue about it, if you have any comments about this question, please write them in issue CALCITE-1515.
Best regards,
Anton Mushin
-----Original Message-----
From: Julian Hyde [mailto:jhyde@apache.org]
Sent: Wednesday, November 30, 2016 12:31 PM
To: dev@calcite.apache.org
Subject: Re: Convert RexSubQuery to RexNode
Adding a TableFunctionScanFactory would be part of it. But also adding a functionScan method, analogous to the scan method but for table functions.
On Wed, Nov 30, 2016 at 12:16 AM, Anton Mushin <An...@epam.com> wrote:
> Hi Julian,
>> RelBuilder does not support TableFunctionScan yet. Can you please log
>> a JIRA case for that? (And implement it, if you are so inclined.)
>
> Do you mean add In RelBuilder TableFunctionScanFactory which will create LogicalTableFunctionScan like as for LogicalTableScan [1] and factory for others RelNodes?
>
> [1]
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/a
> pache/calcite/tools/RelBuilder.java#L125
>
> -----Original Message-----
> From: Julian Hyde [mailto:jhyde@apache.org]
> Sent: Monday, November 28, 2016 8:45 PM
> To: dev@calcite.apache.org
> Subject: Re: Convert RexSubQuery to RexNode
>
> RelBuilder.call deals with a scalar function, and I think what you
> need is a table function. (In java both are just functions, but in
> SQL, or relational algebra, they are different beasts.)
>
> Calcite has a TableFunctionScan, which is a RelNode, takes a function, 0 or more RelNode inputs, and 0 or more other arguments.
>
> RelBuilder does not support TableFunctionScan yet. Can you please log
> a JIRA case for that? (And implement it, if you are so inclined.)
>
> You don’t need to wait for support to be added to RelBuilder. You can create a TableFunctionScan manually, and send it to RelBuilder.push().
>
> Julian
>
>
>> On Nov 28, 2016, at 4:52 AM, Nikolay Vasilishin <Ni...@epam.com> wrote:
>>
>> Hi, devs.
>> I'm trying to implement RexVisitor#visitSubquery method for Flink Table API. Can you help me with a problem?
>>
>>
>> 1. I have to construct a RexNode while having a RexNode with left side of IN (name of column) and RelNode, which stands for table or subquery on the right side of IN operator. If I'm using RexSubQuery.in(rel, rex), I'm failing with exception:
>>
>>
>>
>> java.lang.ClassCastException: org.apache.calcite.rex.RexSubQuery
>> cannot be cast to org.apache.calcite.rex.RexLocalRef
>>
>>
>>
>> at
>> org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuil
>> d
>> er.java:291)
>>
>> at
>> org.apache.calcite.rex.RexProgramBuilder.addCondition(RexProgramBuild
>> e
>> r.java:265)
>>
>> at
>> org.apache.calcite.rel.rules.FilterToCalcRule.onMatch(FilterToCalcRul
>> e
>> .java:66)
>>
>> at
>> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCa
>> l
>> l.java:213)
>>
>> at
>> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPla
>> n
>> ner.java:819)
>>
>> at
>> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:33
>> 4
>> )
>>
>> at
>> org.apache.flink.api.table.BatchTableEnvironment.optimize(BatchTableE
>> n
>> vironment.scala:252)
>>
>> at
>> org.apache.flink.api.table.BatchTableEnvironment.translate(BatchTable
>> E
>> nvironment.scala:287)
>>
>> at
>> org.apache.flink.api.scala.table.BatchTableEnvironment.toDataSet(Batc
>> h
>> TableEnvironment.scala:139)
>>
>> at
>> org.apache.flink.api.scala.table.TableConversions.toDataSet(TableConv
>> e
>> rsions.scala:41)
>>
>> at
>> org.apache.flink.api.scala.batch.table.InITCase.testInSubquery(InITCa
>> s
>> e.scala:72)
>>
>>
>>
>> RelBuilder#call doesn't take RelNode as parameter. Combining RelBuilder#call with output of RexSubQuery.in(rel, rex) as RexNode causes duplicated IN operator in logical plan:
>>
>>
>>
>> LogicalFilter(condition=[IN(IN($2, {
>>
>> LogicalProject(c=[$2])
>>
>> LogicalFilter(condition=[=($1, 6)])
>>
>> LogicalTableScan(table=[[_DataSetTable_0]])
>>
>> }))])
>>
>> LogicalTableScan(table=[[_DataSetTable_0]])
>>
>>
>>
>> Besides, these classes look very similar.
>> So, the question is how to construct proper IN RexNode or how to cast from RexSubQuery to RexNode.
>>
>> Thanks in advance,
>>
>> Nikolay Vasilishin
>> Junior Software Engineer
>>
>> Office: +7 812 611 10 94<tel:+7%20812%20611%2010%2094> x 61997<tel:61997> Cell: +7 921 403 78 55<tel:+7%20921%20403%2078%2055> Email: nikolay_vasilishin@epam.com<ma...@epam.com>
>> Saint-Petersburg, Russia (GMT+3) epam.com<http://www.epam.com>
>>
>> CONFIDENTIALITY CAUTION AND DISCLAIMER This message is intended only
>> for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
>>
>
Re: Convert RexSubQuery to RexNode
Posted by Julian Hyde <jh...@apache.org>.
Adding a TableFunctionScanFactory would be part of it. But also adding
a functionScan method, analogous to the scan method but for table
functions.
On Wed, Nov 30, 2016 at 12:16 AM, Anton Mushin <An...@epam.com> wrote:
> Hi Julian,
>> RelBuilder does not support TableFunctionScan yet. Can you please log a JIRA case for that? (And implement it, if you are so inclined.)
>
> Do you mean add In RelBuilder TableFunctionScanFactory which will create LogicalTableFunctionScan like as for LogicalTableScan [1] and factory for others RelNodes?
>
> [1] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L125
>
> -----Original Message-----
> From: Julian Hyde [mailto:jhyde@apache.org]
> Sent: Monday, November 28, 2016 8:45 PM
> To: dev@calcite.apache.org
> Subject: Re: Convert RexSubQuery to RexNode
>
> RelBuilder.call deals with a scalar function, and I think what you need is a table function. (In java both are just functions, but in SQL, or relational algebra, they are different beasts.)
>
> Calcite has a TableFunctionScan, which is a RelNode, takes a function, 0 or more RelNode inputs, and 0 or more other arguments.
>
> RelBuilder does not support TableFunctionScan yet. Can you please log a JIRA case for that? (And implement it, if you are so inclined.)
>
> You don’t need to wait for support to be added to RelBuilder. You can create a TableFunctionScan manually, and send it to RelBuilder.push().
>
> Julian
>
>
>> On Nov 28, 2016, at 4:52 AM, Nikolay Vasilishin <Ni...@epam.com> wrote:
>>
>> Hi, devs.
>> I'm trying to implement RexVisitor#visitSubquery method for Flink Table API. Can you help me with a problem?
>>
>>
>> 1. I have to construct a RexNode while having a RexNode with left side of IN (name of column) and RelNode, which stands for table or subquery on the right side of IN operator. If I'm using RexSubQuery.in(rel, rex), I'm failing with exception:
>>
>>
>>
>> java.lang.ClassCastException: org.apache.calcite.rex.RexSubQuery
>> cannot be cast to org.apache.calcite.rex.RexLocalRef
>>
>>
>>
>> at
>> org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuild
>> er.java:291)
>>
>> at
>> org.apache.calcite.rex.RexProgramBuilder.addCondition(RexProgramBuilde
>> r.java:265)
>>
>> at
>> org.apache.calcite.rel.rules.FilterToCalcRule.onMatch(FilterToCalcRule
>> .java:66)
>>
>> at
>> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCal
>> l.java:213)
>>
>> at
>> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlan
>> ner.java:819)
>>
>> at
>> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:334
>> )
>>
>> at
>> org.apache.flink.api.table.BatchTableEnvironment.optimize(BatchTableEn
>> vironment.scala:252)
>>
>> at
>> org.apache.flink.api.table.BatchTableEnvironment.translate(BatchTableE
>> nvironment.scala:287)
>>
>> at
>> org.apache.flink.api.scala.table.BatchTableEnvironment.toDataSet(Batch
>> TableEnvironment.scala:139)
>>
>> at
>> org.apache.flink.api.scala.table.TableConversions.toDataSet(TableConve
>> rsions.scala:41)
>>
>> at
>> org.apache.flink.api.scala.batch.table.InITCase.testInSubquery(InITCas
>> e.scala:72)
>>
>>
>>
>> RelBuilder#call doesn't take RelNode as parameter. Combining RelBuilder#call with output of RexSubQuery.in(rel, rex) as RexNode causes duplicated IN operator in logical plan:
>>
>>
>>
>> LogicalFilter(condition=[IN(IN($2, {
>>
>> LogicalProject(c=[$2])
>>
>> LogicalFilter(condition=[=($1, 6)])
>>
>> LogicalTableScan(table=[[_DataSetTable_0]])
>>
>> }))])
>>
>> LogicalTableScan(table=[[_DataSetTable_0]])
>>
>>
>>
>> Besides, these classes look very similar.
>> So, the question is how to construct proper IN RexNode or how to cast from RexSubQuery to RexNode.
>>
>> Thanks in advance,
>>
>> Nikolay Vasilishin
>> Junior Software Engineer
>>
>> Office: +7 812 611 10 94<tel:+7%20812%20611%2010%2094> x 61997<tel:61997> Cell: +7 921 403 78 55<tel:+7%20921%20403%2078%2055> Email: nikolay_vasilishin@epam.com<ma...@epam.com>
>> Saint-Petersburg, Russia (GMT+3) epam.com<http://www.epam.com>
>>
>> CONFIDENTIALITY CAUTION AND DISCLAIMER This message is intended only
>> for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
>>
>
RE: Convert RexSubQuery to RexNode
Posted by Anton Mushin <An...@epam.com>.
Hi Julian,
> RelBuilder does not support TableFunctionScan yet. Can you please log a JIRA case for that? (And implement it, if you are so inclined.)
Do you mean add In RelBuilder TableFunctionScanFactory which will create LogicalTableFunctionScan like as for LogicalTableScan [1] and factory for others RelNodes?
[1] https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L125
-----Original Message-----
From: Julian Hyde [mailto:jhyde@apache.org]
Sent: Monday, November 28, 2016 8:45 PM
To: dev@calcite.apache.org
Subject: Re: Convert RexSubQuery to RexNode
RelBuilder.call deals with a scalar function, and I think what you need is a table function. (In java both are just functions, but in SQL, or relational algebra, they are different beasts.)
Calcite has a TableFunctionScan, which is a RelNode, takes a function, 0 or more RelNode inputs, and 0 or more other arguments.
RelBuilder does not support TableFunctionScan yet. Can you please log a JIRA case for that? (And implement it, if you are so inclined.)
You don’t need to wait for support to be added to RelBuilder. You can create a TableFunctionScan manually, and send it to RelBuilder.push().
Julian
> On Nov 28, 2016, at 4:52 AM, Nikolay Vasilishin <Ni...@epam.com> wrote:
>
> Hi, devs.
> I'm trying to implement RexVisitor#visitSubquery method for Flink Table API. Can you help me with a problem?
>
>
> 1. I have to construct a RexNode while having a RexNode with left side of IN (name of column) and RelNode, which stands for table or subquery on the right side of IN operator. If I'm using RexSubQuery.in(rel, rex), I'm failing with exception:
>
>
>
> java.lang.ClassCastException: org.apache.calcite.rex.RexSubQuery
> cannot be cast to org.apache.calcite.rex.RexLocalRef
>
>
>
> at
> org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuild
> er.java:291)
>
> at
> org.apache.calcite.rex.RexProgramBuilder.addCondition(RexProgramBuilde
> r.java:265)
>
> at
> org.apache.calcite.rel.rules.FilterToCalcRule.onMatch(FilterToCalcRule
> .java:66)
>
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCal
> l.java:213)
>
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlan
> ner.java:819)
>
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:334
> )
>
> at
> org.apache.flink.api.table.BatchTableEnvironment.optimize(BatchTableEn
> vironment.scala:252)
>
> at
> org.apache.flink.api.table.BatchTableEnvironment.translate(BatchTableE
> nvironment.scala:287)
>
> at
> org.apache.flink.api.scala.table.BatchTableEnvironment.toDataSet(Batch
> TableEnvironment.scala:139)
>
> at
> org.apache.flink.api.scala.table.TableConversions.toDataSet(TableConve
> rsions.scala:41)
>
> at
> org.apache.flink.api.scala.batch.table.InITCase.testInSubquery(InITCas
> e.scala:72)
>
>
>
> RelBuilder#call doesn't take RelNode as parameter. Combining RelBuilder#call with output of RexSubQuery.in(rel, rex) as RexNode causes duplicated IN operator in logical plan:
>
>
>
> LogicalFilter(condition=[IN(IN($2, {
>
> LogicalProject(c=[$2])
>
> LogicalFilter(condition=[=($1, 6)])
>
> LogicalTableScan(table=[[_DataSetTable_0]])
>
> }))])
>
> LogicalTableScan(table=[[_DataSetTable_0]])
>
>
>
> Besides, these classes look very similar.
> So, the question is how to construct proper IN RexNode or how to cast from RexSubQuery to RexNode.
>
> Thanks in advance,
>
> Nikolay Vasilishin
> Junior Software Engineer
>
> Office: +7 812 611 10 94<tel:+7%20812%20611%2010%2094> x 61997<tel:61997> Cell: +7 921 403 78 55<tel:+7%20921%20403%2078%2055> Email: nikolay_vasilishin@epam.com<ma...@epam.com>
> Saint-Petersburg, Russia (GMT+3) epam.com<http://www.epam.com>
>
> CONFIDENTIALITY CAUTION AND DISCLAIMER This message is intended only
> for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
>
Re: Convert RexSubQuery to RexNode
Posted by Julian Hyde <jh...@apache.org>.
RelBuilder.call deals with a scalar function, and I think what you need is a table function. (In java both are just functions, but in SQL, or relational algebra, they are different beasts.)
Calcite has a TableFunctionScan, which is a RelNode, takes a function, 0 or more RelNode inputs, and 0 or more other arguments.
RelBuilder does not support TableFunctionScan yet. Can you please log a JIRA case for that? (And implement it, if you are so inclined.)
You don’t need to wait for support to be added to RelBuilder. You can create a TableFunctionScan manually, and send it to RelBuilder.push().
Julian
> On Nov 28, 2016, at 4:52 AM, Nikolay Vasilishin <Ni...@epam.com> wrote:
>
> Hi, devs.
> I'm trying to implement RexVisitor#visitSubquery method for Flink Table API. Can you help me with a problem?
>
>
> 1. I have to construct a RexNode while having a RexNode with left side of IN (name of column) and RelNode, which stands for table or subquery on the right side of IN operator. If I'm using RexSubQuery.in(rel, rex), I'm failing with exception:
>
>
>
> java.lang.ClassCastException: org.apache.calcite.rex.RexSubQuery cannot be cast to org.apache.calcite.rex.RexLocalRef
>
>
>
> at org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuilder.java:291)
>
> at org.apache.calcite.rex.RexProgramBuilder.addCondition(RexProgramBuilder.java:265)
>
> at org.apache.calcite.rel.rules.FilterToCalcRule.onMatch(FilterToCalcRule.java:66)
>
> at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:213)
>
> at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:819)
>
> at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:334)
>
> at org.apache.flink.api.table.BatchTableEnvironment.optimize(BatchTableEnvironment.scala:252)
>
> at org.apache.flink.api.table.BatchTableEnvironment.translate(BatchTableEnvironment.scala:287)
>
> at org.apache.flink.api.scala.table.BatchTableEnvironment.toDataSet(BatchTableEnvironment.scala:139)
>
> at org.apache.flink.api.scala.table.TableConversions.toDataSet(TableConversions.scala:41)
>
> at org.apache.flink.api.scala.batch.table.InITCase.testInSubquery(InITCase.scala:72)
>
>
>
> RelBuilder#call doesn't take RelNode as parameter. Combining RelBuilder#call with output of RexSubQuery.in(rel, rex) as RexNode causes duplicated IN operator in logical plan:
>
>
>
> LogicalFilter(condition=[IN(IN($2, {
>
> LogicalProject(c=[$2])
>
> LogicalFilter(condition=[=($1, 6)])
>
> LogicalTableScan(table=[[_DataSetTable_0]])
>
> }))])
>
> LogicalTableScan(table=[[_DataSetTable_0]])
>
>
>
> Besides, these classes look very similar.
> So, the question is how to construct proper IN RexNode or how to cast from RexSubQuery to RexNode.
>
> Thanks in advance,
>
> Nikolay Vasilishin
> Junior Software Engineer
>
> Office: +7 812 611 10 94<tel:+7%20812%20611%2010%2094> x 61997<tel:61997> Cell: +7 921 403 78 55<tel:+7%20921%20403%2078%2055> Email: nikolay_vasilishin@epam.com<ma...@epam.com>
> Saint-Petersburg, Russia (GMT+3) epam.com<http://www.epam.com>
>
> CONFIDENTIALITY CAUTION AND DISCLAIMER
> This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
>