You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Bhavya Aggarwal <bh...@knoldus.com> on 2020/11/26 07:50:05 UTC
How to convert a logical operator into multiple physical operators
Hi All,
We are trying to use Apache Calcite as and CBO for our homegrown DB and
there one Logical operator will translate into multiple Physical Operations
at our database e.g. a simple TableScan will convert into the given
structure. In the example above we have to decompress the data, then
rechunk it. However, I am able to create 1:1 conversion. Please let me know
how to proceed. Any help will be much appreciated.
select id from users where id = 5;
AddToNetworkQueue 1
Rechunk 2
ReorderInput 3
ReorderInput 4
Filter 5
Rechunk 6
Decompress 7
ReadTable 8
Thanks and regards
Bhavya
--
*Bhavya Aggarwal*
CTO & Partner
Knoldus Inc. <http://www.knoldus.com/>
+91-9910483067
Canada - USA - India - Singapore
<https://in.linkedin.com/company/knoldus> <https://twitter.com/Knolspeak>
<https://www.facebook.com/KnoldusSoftware/> <https://blog.knoldus.com/>
--
Your feedback matters - At Knoldus we aim to be very professional in our
quality of work, commitment to results, and proactive communication. If you
feel otherwise please share your feedback
<https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work on it.
Re: How to convert a logical operator into multiple physical operators
Posted by Chunwei Lei <ch...@gmail.com>.
Hi, Bhavya.
I think all you need to do is to create the node correctly. You can
refer to how the EnumerableMergeJoin is created[1].
[1]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableJoinRule.java#L86
Best,
Chunwei
On Fri, Nov 27, 2020 at 9:23 AM Michael Mior <mm...@apache.org> wrote:
> Do you need to use CBO for the physical operators as well? If not, you
> could certainly just keep the single TableScan and perform whatever
> operations you want at query execution time. If you want to use CBO
> with your physical operators, there is no requirement that an
> optimizer rule produce the same number of RelNodes. You could write a
> rule which converts a TableScan into the tree of operators you showed
> and then write further rules which manipulate the physical operators.
> --
> Michael Mior
> mmior@apache.org
>
> Le jeu. 26 nov. 2020 à 16:53, Bhavya Aggarwal <bh...@knoldus.com> a
> écrit :
> >
> > Hi All,
> >
> > We are trying to use Apache Calcite as and CBO for our homegrown DB and
> > there one Logical operator will translate into multiple Physical
> Operations
> > at our database e.g. a simple TableScan will convert into the given
> > structure. In the example above we have to decompress the data, then
> > rechunk it. However, I am able to create 1:1 conversion. Please let me
> know
> > how to proceed. Any help will be much appreciated.
> >
> > select id from users where id = 5;
> > AddToNetworkQueue 1
> > Rechunk 2
> > ReorderInput 3
> > ReorderInput 4
> > Filter 5
> > Rechunk 6
> > Decompress 7
> > ReadTable 8
> >
> > Thanks and regards
> > Bhavya
> > --
> > *Bhavya Aggarwal*
> > CTO & Partner
> > Knoldus Inc. <http://www.knoldus.com/>
> > +91-9910483067
> > Canada - USA - India - Singapore
> > <https://in.linkedin.com/company/knoldus> <https://twitter.com/Knolspeak
> >
> > <https://www.facebook.com/KnoldusSoftware/> <https://blog.knoldus.com/>
> >
> > --
> > Your feedback matters - At Knoldus we aim to be very professional in our
> > quality of work, commitment to results, and proactive communication. If
> you
> > feel otherwise please share your feedback
> > <https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work on it.
>
Re: How to convert a logical operator into multiple physical operators
Posted by Michael Mior <mm...@apache.org>.
Do you need to use CBO for the physical operators as well? If not, you
could certainly just keep the single TableScan and perform whatever
operations you want at query execution time. If you want to use CBO
with your physical operators, there is no requirement that an
optimizer rule produce the same number of RelNodes. You could write a
rule which converts a TableScan into the tree of operators you showed
and then write further rules which manipulate the physical operators.
--
Michael Mior
mmior@apache.org
Le jeu. 26 nov. 2020 à 16:53, Bhavya Aggarwal <bh...@knoldus.com> a écrit :
>
> Hi All,
>
> We are trying to use Apache Calcite as and CBO for our homegrown DB and
> there one Logical operator will translate into multiple Physical Operations
> at our database e.g. a simple TableScan will convert into the given
> structure. In the example above we have to decompress the data, then
> rechunk it. However, I am able to create 1:1 conversion. Please let me know
> how to proceed. Any help will be much appreciated.
>
> select id from users where id = 5;
> AddToNetworkQueue 1
> Rechunk 2
> ReorderInput 3
> ReorderInput 4
> Filter 5
> Rechunk 6
> Decompress 7
> ReadTable 8
>
> Thanks and regards
> Bhavya
> --
> *Bhavya Aggarwal*
> CTO & Partner
> Knoldus Inc. <http://www.knoldus.com/>
> +91-9910483067
> Canada - USA - India - Singapore
> <https://in.linkedin.com/company/knoldus> <https://twitter.com/Knolspeak>
> <https://www.facebook.com/KnoldusSoftware/> <https://blog.knoldus.com/>
>
> --
> Your feedback matters - At Knoldus we aim to be very professional in our
> quality of work, commitment to results, and proactive communication. If you
> feel otherwise please share your feedback
> <https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work on it.
Re: How to convert a logical operator into multiple physical operators
Posted by Bhavya Aggarwal <bh...@knoldus.com>.
Thanks for the reply, so I looked at the example and created a Nested
RelNode for my TableScanOperator as shown below
override def onMatch(call: RelOptRuleCall): Unit = {
val tableScan : LogicalTableScan = call.rel[LogicalTableScan](0);
val builder = call.builder()
val newOp = new ReadTable(
tableScan.getCluster,
RelTraitSet.createEmpty
.plus(SqreamRel.CONVENTION),
List.empty,
tableScan.getTable
)
val input = builder.push(tableScan).empty().build()
val cpuDecompress = new CpuDecompress(tableScan.getCluster,
RelTraitSet.createEmpty
.plus(SqreamRel.CONVENTION), newOp)
val collationListBuilder = new ImmutableList.Builder[RelFieldCollation]
val cpuToGpu = new CpuToGpu(tableScan.getCluster,
RelTraitSet.createEmpty().plus(SqreamRel.CONVENTION),
cpuDecompress,RelCollationImpl.of(collationListBuilder.build()))
val gpuTransform = new
GpuTransform(tableScan.getCluster,RelTraitSet.createEmpty().plus(SqreamRel.CONVENTION),
cpuToGpu )
builder.push(gpuTransform)
call.transformTo( builder.build())
}
But in this case, I am getting an exception from the Planner which is
related to ReadTable which is extending the tablescan. How can I wrap my
operators around the table scan operator.
Exception in thread "main" java.lang.ClassCastException: class
com.sqream.calcite.aux.rel.ReadTable cannot be cast to class
org.apache.calcite.plan.hep.HepRelVertex
(com.sqream.calcite.aux.rel.ReadTable and
org.apache.calcite.plan.hep.HepRelVertex are in unnamed module of loader
'app')
at org.apache.calcite.plan.hep.HepPlanner.addRelToGraph(HepPlanner.java:849)
at org.apache.calcite.plan.hep.HepPlanner.addRelToGraph(HepPlanner.java:819)
at org.apache.calcite.plan.hep.HepPlanner.addRelToGraph(HepPlanner.java:819)
at
org.apache.calcite.plan.hep.HepPlanner.applyTransformationResults(HepPlanner.java:754)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:427)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:264)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:223)
at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:210)
at TestSqream$.delayedEndpoint$TestSqream$1(TestSqream.scala:77)
at TestSqream$delayedInit$body.apply(TestSqream.scala:24)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1$adapted(App.scala:80)
at scala.collection.immutable.List.foreach(List.scala:431)
at scala.App.main(App.scala:80)
at scala.App.main$(App.scala:78)
at TestSqream$.main(TestSqream.scala:24)
at TestSqream.main(TestSqream.scala)
>
>
> On Thu, Nov 26, 2020 at 1:20 PM Bhavya Aggarwal <bh...@knoldus.com>
> wrote:
>
>> Hi All,
>>
>> We are trying to use Apache Calcite as and CBO for our homegrown DB and
>> there one Logical operator will translate into multiple Physical Operations
>> at our database e.g. a simple TableScan will convert into the given
>> structure. In the example above we have to decompress the data, then
>> rechunk it. However, I am able to create 1:1 conversion. Please let me know
>> how to proceed. Any help will be much appreciated.
>>
>> select id from users where id = 5;
>> AddToNetworkQueue 1
>> Rechunk 2
>> ReorderInput 3
>> ReorderInput 4
>> Filter 5
>> Rechunk 6
>> Decompress 7
>> ReadTable 8
>>
>> Thanks and regards
>> Bhavya
>> --
>> *Bhavya Aggarwal*
>> CTO & Partner
>> Knoldus Inc. <http://www.knoldus.com/>
>> +91-9910483067
>> Canada - USA - India - Singapore
>> <https://in.linkedin.com/company/knoldus> <https://twitter.com/Knolspeak>
>> <https://www.facebook.com/KnoldusSoftware/> <https://blog.knoldus.com/>
>>
>
>
>
--
Your feedback matters - At Knoldus we aim to be very professional in our
quality of work, commitment to results, and proactive communication. If you
feel otherwise please share your feedback
<https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work on it.