What is the best planner for an Adapter

[Marieke Gueye <ma...@google.com.INVALID>]

Hey everybody,

I created an adapter with some rules pushing down everything (Projects,
Filter,..)  to a single Rel, let's call it CustomRel:

So for example my Project Rule basically transforms:
LogicalProject(R_NAME=[$0])
  CustomRel(projects=[[$0, $1]])
to
  CustomRel(projects=[[$0]])

I also have a CustomRelToEnumerableConverter &
CustomRelToEnumerableConverterRule that converts the trait set from my
custom convention (in the CustomRel) to an Enumerable.

I am using Avatica with my adapter and realised that the Volcano planner is
not working as I would expect it to. Instead of first pushing down
everything to CustomRel then converting to enumerable, it is pushing down
only one rel above it, applying the convert and then doing some
EnumerableCalc.. Changing the selfCost of CustomRel did not help.

I also tried overriding the planner (overriding createPlanner from
CalcitePrepareImpl) so it uses a HepPlanner but it looks like the planner
loses the trait information and the converter is never fired.

Any idea what would be the solution in this case?

Thanks for your help

Re: What is the best planner for an Adapter

[Marieke Gueye <ma...@google.com.INVALID>]

Thank you so much I will check it out!

On Sun, Apr 24, 2022 at 5:21 AM Stamatis Zampetakis <za...@gmail.com>
wrote:

> Hi Marieke,
>
> As Benchao already said in many use cases you will find both planners to be
> in use.
>
> Although the general planner interface is the same the two planners have
> some distinctive characteristics so they are not functionally equivalent.
>
> The Volcano planner apart from a bunch of rules in the input it also allows
> you to request specific properties (traits) to be satisfied in the output
> plan. If the properties cannot be satisfied then you will get a
> CannotPlanException.
>
> In contrast with Volcano, the heuristic planner does not enforce traits so
> you can obtain "invalid" (not having the desired traits) plans and no
> exceptions. Most of the existing rules that target specific conventions
> (e.g Enumerable rules) assume that Volcano is in use and may not work well
> if you use the heuristic planner.
>
> In an classic Calcite adapter, which usually handles multiple conventions,
> you will probably need the Volcano planner.
>
> The use case you described, pushing everything to a single relation, seems
> very similar to how the Druid adapter works so you can have a look there
> for some inspiration.
>
> Best,
> Stamatis
>
> On Sun, Apr 24, 2022, 7:04 AM Benchao Li <li...@apache.org> wrote:
>
> > Hi Marieke,
> >
> > Usually for a fledged SQL engine built on top of Calcite, it uses both
> > planner.
> > And we have Program[1] abstraction to organize multi stages to plan a
> > query.
> > You can take a look at Calcite's default implementation here[2].
> >
> > [1]
> >
> >
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/Program.java
> > [2]
> >
> >
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L176
> >
> > Marieke Gueye <ma...@google.com.invalid> 于2022年4月23日周六 04:50写道：
> >
> > > Hey everybody,
> > >
> > > I created an adapter with some rules pushing down everything (Projects,
> > > Filter,..)  to a single Rel, let's call it CustomRel:
> > >
> > > So for example my Project Rule basically transforms:
> > > LogicalProject(R_NAME=[$0])
> > >   CustomRel(projects=[[$0, $1]])
> > > to
> > >   CustomRel(projects=[[$0]])
> > >
> > > I also have a CustomRelToEnumerableConverter &
> > > CustomRelToEnumerableConverterRule that converts the trait set from my
> > > custom convention (in the CustomRel) to an Enumerable.
> > >
> > > I am using Avatica with my adapter and realised that the Volcano
> planner
> > is
> > > not working as I would expect it to. Instead of first pushing down
> > > everything to CustomRel then converting to enumerable, it is pushing
> down
> > > only one rel above it, applying the convert and then doing some
> > > EnumerableCalc.. Changing the selfCost of CustomRel did not help.
> > >
> > > I also tried overriding the planner (overriding createPlanner from
> > > CalcitePrepareImpl) so it uses a HepPlanner but it looks like the
> planner
> > > loses the trait information and the converter is never fired.
> > >
> > > Any idea what would be the solution in this case?
> > >
> > > Thanks for your help
> > >
> >
> >
> > --
> >
> > Best,
> > Benchao Li
> >
>

Re: What is the best planner for an Adapter

[Stamatis Zampetakis <za...@gmail.com>]

Hi Marieke,

As Benchao already said in many use cases you will find both planners to be
in use.

Although the general planner interface is the same the two planners have
some distinctive characteristics so they are not functionally equivalent.

The Volcano planner apart from a bunch of rules in the input it also allows
you to request specific properties (traits) to be satisfied in the output
plan. If the properties cannot be satisfied then you will get a
CannotPlanException.

In contrast with Volcano, the heuristic planner does not enforce traits so
you can obtain "invalid" (not having the desired traits) plans and no
exceptions. Most of the existing rules that target specific conventions
(e.g Enumerable rules) assume that Volcano is in use and may not work well
if you use the heuristic planner.

In an classic Calcite adapter, which usually handles multiple conventions,
you will probably need the Volcano planner.

The use case you described, pushing everything to a single relation, seems
very similar to how the Druid adapter works so you can have a look there
for some inspiration.

Best,
Stamatis

On Sun, Apr 24, 2022, 7:04 AM Benchao Li <li...@apache.org> wrote:

> Hi Marieke,
>
> Usually for a fledged SQL engine built on top of Calcite, it uses both
> planner.
> And we have Program[1] abstraction to organize multi stages to plan a
> query.
> You can take a look at Calcite's default implementation here[2].
>
> [1]
>
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/Program.java
> [2]
>
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L176
>
> Marieke Gueye <ma...@google.com.invalid> 于2022年4月23日周六 04:50写道：
>
> > Hey everybody,
> >
> > I created an adapter with some rules pushing down everything (Projects,
> > Filter,..)  to a single Rel, let's call it CustomRel:
> >
> > So for example my Project Rule basically transforms:
> > LogicalProject(R_NAME=[$0])
> >   CustomRel(projects=[[$0, $1]])
> > to
> >   CustomRel(projects=[[$0]])
> >
> > I also have a CustomRelToEnumerableConverter &
> > CustomRelToEnumerableConverterRule that converts the trait set from my
> > custom convention (in the CustomRel) to an Enumerable.
> >
> > I am using Avatica with my adapter and realised that the Volcano planner
> is
> > not working as I would expect it to. Instead of first pushing down
> > everything to CustomRel then converting to enumerable, it is pushing down
> > only one rel above it, applying the convert and then doing some
> > EnumerableCalc.. Changing the selfCost of CustomRel did not help.
> >
> > I also tried overriding the planner (overriding createPlanner from
> > CalcitePrepareImpl) so it uses a HepPlanner but it looks like the planner
> > loses the trait information and the converter is never fired.
> >
> > Any idea what would be the solution in this case?
> >
> > Thanks for your help
> >
>
>
> --
>
> Best,
> Benchao Li
>

Re: What is the best planner for an Adapter

[Benchao Li <li...@apache.org>]

Hi Marieke,

Usually for a fledged SQL engine built on top of Calcite, it uses both
planner.
And we have Program[1] abstraction to organize multi stages to plan a query.
You can take a look at Calcite's default implementation here[2].

[1]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/tools/Program.java
[2]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L176

Marieke Gueye <ma...@google.com.invalid> 于2022年4月23日周六 04:50写道：

> Hey everybody,
>
> I created an adapter with some rules pushing down everything (Projects,
> Filter,..)  to a single Rel, let's call it CustomRel:
>
> So for example my Project Rule basically transforms:
> LogicalProject(R_NAME=[$0])
>   CustomRel(projects=[[$0, $1]])
> to
>   CustomRel(projects=[[$0]])
>
> I also have a CustomRelToEnumerableConverter &
> CustomRelToEnumerableConverterRule that converts the trait set from my
> custom convention (in the CustomRel) to an Enumerable.
>
> I am using Avatica with my adapter and realised that the Volcano planner is
> not working as I would expect it to. Instead of first pushing down
> everything to CustomRel then converting to enumerable, it is pushing down
> only one rel above it, applying the convert and then doing some
> EnumerableCalc.. Changing the selfCost of CustomRel did not help.
>
> I also tried overriding the planner (overriding createPlanner from
> CalcitePrepareImpl) so it uses a HepPlanner but it looks like the planner
> loses the trait information and the converter is never fired.
>
> Any idea what would be the solution in this case?
>
> Thanks for your help
>


-- 

Best,
Benchao Li