Plug a new cost model to Calcite

[Γιώργος Θεοδωράκης <gi...@gmail.com>]

Hello,

I think I have understood the basics about RelOptCost, transformer and
converter rules and how to use them in Volcano in order to set my own
cost-model, from examples I have seen in Drill and Hive mainly.
As I am thinking about t it right now, I should:
1)define my cost model and optimization goal (Volcano uses Lt method) in a
CustomCostBase class that implements RelOptCost
2)define my operators with the same logic as they are and override their
computeCost method
3)define my proper converter and transform rules

However, in order to use Volcano, I have to use some rules with Convention
that is not NONE. So I think I have to implement my own custom Convention
that my rules should have and convert the logical rules to my custom ones.
Then enforce my rules on these custom rules and get an optimized plan.

Do I miss something in the logic I describe above? Do you have any
suggestions on how to do it in another way?

Thank you in advance,
George

Re: Plug a new cost model to Calcite

[Γιώργος Θεοδωράκης <gi...@gmail.com>]

I had to set the convention right like this before using the transform
method:
    RelTraitSet traitSet =
planner.getEmptyTraitSet().replace(SaberRel.SABER_LOGICAL);
    RelNode volcanoPlan = planner.transform(0, traitSet, convertedNode);


2016-12-22 16:48 GMT+02:00 Γιώργος Θεοδωράκης <gi...@gmail.com>:

> I have written two conversion rules to test my custom TableScan and
> Project on Volcano (I use as Convention: new Convention.Impl("LOGICAL",
> SaberRel.class)). To begin with, I want only to convert the initial logical
> rules of Calcite to my custom logical rules. I have added to my planner the
> two conversion rules with AbstractConverter.ExpandConversionRule.INSTANCE.
> However, I get this error:
>
> Exception in thread "main" org.apache.calcite.plan.RelOptPlanner$CannotPlanException:
> Node [rel#8:Subset#0.ENUMERABLE.[]] could not be implemented; planner state:
>
> Root: rel#8:Subset#0.ENUMERABLE.[]
> Original rel:
>
> Sets:
> Set#0, type: RecordType(TIMESTAMP(0) rowtime, INTEGER productid, INTEGER
> description)
> rel#4:Subset#0.NONE.[0], best=null, importance=0.0
> rel#2:LogicalTableScan.NONE.[[0]](table=[s, products]), rowcount=16384.0,
> cumulative cost={inf}
> rel#5:LogicalProject.NONE.[[0]](input=rel#4:Subset#0.NONE.[
> 0],rowtime=$0,productid=$1,description=$2), rowcount=16384.0, cumulative
> cost={inf}
> rel#8:Subset#0.ENUMERABLE.[], best=null, importance=0.0
> rel#10:Subset#0.LOGICAL.[0], best=rel#9, importance=0.0
> rel#9:SaberTableScanRel.LOGICAL.[[0]](table=[s, products]),
> rowcount=16384.0, cumulative cost={16384.0 rows, 16385.0 cpu, 0.0 io,
> 16384.0 rate, 0.0 memory, 0.0 window, 0.0 network}
> rel#11:SaberProjectRel.LOGICAL.[[0]](input=rel#10:
> Subset#0.LOGICAL.[0],rowtime=$0,productid=$1,description=$2),
> rowcount=16384.0, cumulative cost={32768.0 rows, 147457.0 cpu, 0.0 io,
> 16384.0 rate, 0.0 memory, 1.0 window, 0.0 network}
>
>
> at org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit(
> RelSubset.java:443)
> at org.apache.calcite.plan.volcano.RelSubset.buildCheapestPlan(RelSubset.
> java:293)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.
> findBestExp(VolcanoPlanner.java:835)
> at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:334)
> at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:308)
> at calcite.planner.SaberPlanner.getLogicalPlan(SaberPlanner.java:309)
> at calcite.Tester.main(Tester.java:70)
>
> Why does it still try to implement Enumerable convention? When I add the
> Enumerable rules I get the Enumerable logical plan and not my custom. What
> should I do?
>
> 2016-12-17 22:11 GMT+02:00 Γιώργος Θεοδωράκης <gi...@gmail.com>:
>
>> Hello,
>>
>> I think I have understood the basics about RelOptCost, transformer and
>> converter rules and how to use them in Volcano in order to set my own
>> cost-model, from examples I have seen in Drill and Hive mainly.
>> As I am thinking about t it right now, I should:
>> 1)define my cost model and optimization goal (Volcano uses Lt method) in
>> a CustomCostBase class that implements RelOptCost
>> 2)define my operators with the same logic as they are and override their
>> computeCost method
>> 3)define my proper converter and transform rules
>>
>> However, in order to use Volcano, I have to use some rules with
>> Convention that is not NONE. So I think I have to implement my own custom
>> Convention that my rules should have and convert the logical rules to my
>> custom ones. Then enforce my rules on these custom rules and get an
>> optimized plan.
>>
>> Do I miss something in the logic I describe above? Do you have any
>> suggestions on how to do it in another way?
>>
>> Thank you in advance,
>> George
>>
>>
>

Re: Plug a new cost model to Calcite

[Γιώργος Θεοδωράκης <gi...@gmail.com>]

I have written two conversion rules to test my custom TableScan and Project
on Volcano (I use as Convention: new Convention.Impl("LOGICAL",
SaberRel.class)). To begin with, I want only to convert the initial logical
rules of Calcite to my custom logical rules. I have added to my planner the
two conversion rules with AbstractConverter.ExpandConversionRule.INSTANCE.
However, I get this error:

Exception in thread "main"
org.apache.calcite.plan.RelOptPlanner$CannotPlanException: Node
[rel#8:Subset#0.ENUMERABLE.[]] could not be implemented; planner state:

Root: rel#8:Subset#0.ENUMERABLE.[]
Original rel:

Sets:
Set#0, type: RecordType(TIMESTAMP(0) rowtime, INTEGER productid, INTEGER
description)
rel#4:Subset#0.NONE.[0], best=null, importance=0.0
rel#2:LogicalTableScan.NONE.[[0]](table=[s, products]), rowcount=16384.0,
cumulative cost={inf}
rel#5:LogicalProject.NONE.[[0]](input=rel#4:Subset#0.NONE.[0],rowtime=$0,productid=$1,description=$2),
rowcount=16384.0, cumulative cost={inf}
rel#8:Subset#0.ENUMERABLE.[], best=null, importance=0.0
rel#10:Subset#0.LOGICAL.[0], best=rel#9, importance=0.0
rel#9:SaberTableScanRel.LOGICAL.[[0]](table=[s, products]),
rowcount=16384.0, cumulative cost={16384.0 rows, 16385.0 cpu, 0.0 io,
16384.0 rate, 0.0 memory, 0.0 window, 0.0 network}
rel#11:SaberProjectRel.LOGICAL.[[0]](input=rel#10:Subset#0.LOGICAL.[0],rowtime=$0,productid=$1,description=$2),
rowcount=16384.0, cumulative cost={32768.0 rows, 147457.0 cpu, 0.0 io,
16384.0 rate, 0.0 memory, 1.0 window, 0.0 network}


at
org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit(RelSubset.java:443)
at
org.apache.calcite.plan.volcano.RelSubset.buildCheapestPlan(RelSubset.java:293)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:835)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:334)
at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:308)
at calcite.planner.SaberPlanner.getLogicalPlan(SaberPlanner.java:309)
at calcite.Tester.main(Tester.java:70)

Why does it still try to implement Enumerable convention? When I add the
Enumerable rules I get the Enumerable logical plan and not my custom. What
should I do?

2016-12-17 22:11 GMT+02:00 Γιώργος Θεοδωράκης <gi...@gmail.com>:

> Hello,
>
> I think I have understood the basics about RelOptCost, transformer and
> converter rules and how to use them in Volcano in order to set my own
> cost-model, from examples I have seen in Drill and Hive mainly.
> As I am thinking about t it right now, I should:
> 1)define my cost model and optimization goal (Volcano uses Lt method) in a
> CustomCostBase class that implements RelOptCost
> 2)define my operators with the same logic as they are and override their
> computeCost method
> 3)define my proper converter and transform rules
>
> However, in order to use Volcano, I have to use some rules with Convention
> that is not NONE. So I think I have to implement my own custom Convention
> that my rules should have and convert the logical rules to my custom ones.
> Then enforce my rules on these custom rules and get an optimized plan.
>
> Do I miss something in the logic I describe above? Do you have any
> suggestions on how to do it in another way?
>
> Thank you in advance,
> George
>
>