You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Meron Avigdor <me...@gigaspaces.com> on 2020/08/02 15:55:02 UTC
Avatica and HepPlanner
Hi.
I have implemented an extension of org.apache.calcite.jdbc.Driver, and I
have the model handler extend org.apache.calcite.avatica.HandlerImpl
For performance optimizations, I would like to test VolcanoPlanner vs.
HepPlanner
as the documentation says that the latter is faster.
How is this to be configured? I could not find a hook point to return the
planner to be used.
Thank you.
Meron
Re: Avatica and HepPlanner
Posted by Enrico Olivelli <eo...@gmail.com>.
Is there some documentation or best practice doc about how using both of
the two planners ?
I guess you have to pass a set of rules to the HepPlanner in order to
perform some simpler steps
and then pass other rules to VolcanoPlanner in order to create the physical
plan.
What is the RelNode to pass from the HepPlanner to the VolcanoPlanner ?
// parse
SqlNode n = planner.parse(query);
// validate
n = planner.validate(n);
// create "logical plan"
RelNode logicalPlan = planner.rel(n).project();
....
// get Volcano from "logicalPlan"
RelOptCluster cluster = logicalPlan.getCluster();
elOptPlanner optPlanner = cluster.getPlanner();
....
// perform physical planning
RelNode bestExp = optPlanner.findBestExp();
Regards
Enrico
Il giorno mar 4 ago 2020 alle ore 08:05 Meron Avigdor <
meron.avigdor@gigaspaces.com> ha scritto:
> Thanks Rui and Stamatis for the clarification. I was probably misled by the
> documentation,
> as it stated that there are 2 implementations for planner rules. Thus I
> figured that this is configurable in some way.
> Also, it was mentioned that the Heuristic approach carried by the
> HepPlanner has less overhead.
>
> If you say the two are complementary to each other, I will approach it
> differently.
> Thank you for the assistance.
>
>
> On Tue, Aug 4, 2020 at 12:50 AM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
> > Hi Meron,
> >
> > The two planners are not equivalent so the comparison is not totally
> fair.
> > Even if you apply the same ruleset it does not mean that you are going
> > to obtain the same plan.
> >
> > The two planners are complementary to each other and they are used in
> > different stages of the optimization process thus it is not possible to
> use
> > exclusively one or the other. For the same reason I think it will be
> tricky
> > to
> > add such configuration.
> >
> > If you notice a performance problem with the existing planners or have
> > ideas to improve them feel free to file a JIRA.
> >
> > Best,
> > Stamatis
> >
> > On Sun, Aug 2, 2020 at 10:38 PM Rui Wang <am...@apache.org> wrote:
> >
> > > I tried to dig into the Calcite codebase and found there is likely no
> way
> > > to enable JDBC codepath:
> > > 1. There is no planner config in CalciteConnectionProperty [1].
> > > 2. In CalcitePrepareImpl, there is only one createPlannerFactory with
> > one
> > > createPlanner, in which VolcanoPlanner is used [2].
> > >
> > >
> > > If my understanding is correct, then there could be an improvement that
> > > adds a config to CalciteConnectionProperty, and use that config to
> > control
> > > which planner to use by JDBC approach.
> > >
> > >
> > > [1]:
> > >
> > >
> >
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/config/CalciteConnectionProperty.java#L37
> > > [2]:
> > >
> > >
> >
> https://github.com/apache/calcite/blob/2088488ac8327b19512a76a122cae2961fc551c3/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L427
> > >
> > >
> > > -Rui
> > >
> > > On Sun, Aug 2, 2020 at 8:55 AM Meron Avigdor <
> > meron.avigdor@gigaspaces.com
> > > >
> > > wrote:
> > >
> > > > Hi.
> > > > I have implemented an extension of org.apache.calcite.jdbc.Driver,
> and
> > I
> > > > have the model handler extend org.apache.calcite.avatica.HandlerImpl
> > > >
> > > > For performance optimizations, I would like to test VolcanoPlanner
> vs.
> > > > HepPlanner
> > > > as the documentation says that the latter is faster.
> > > >
> > > > How is this to be configured? I could not find a hook point to return
> > the
> > > > planner to be used.
> > > >
> > > > Thank you.
> > > > Meron
> > > >
> > >
> >
>
Re: Avatica and HepPlanner
Posted by Meron Avigdor <me...@gigaspaces.com>.
Thanks Rui and Stamatis for the clarification. I was probably misled by the
documentation,
as it stated that there are 2 implementations for planner rules. Thus I
figured that this is configurable in some way.
Also, it was mentioned that the Heuristic approach carried by the
HepPlanner has less overhead.
If you say the two are complementary to each other, I will approach it
differently.
Thank you for the assistance.
On Tue, Aug 4, 2020 at 12:50 AM Stamatis Zampetakis <za...@gmail.com>
wrote:
> Hi Meron,
>
> The two planners are not equivalent so the comparison is not totally fair.
> Even if you apply the same ruleset it does not mean that you are going
> to obtain the same plan.
>
> The two planners are complementary to each other and they are used in
> different stages of the optimization process thus it is not possible to use
> exclusively one or the other. For the same reason I think it will be tricky
> to
> add such configuration.
>
> If you notice a performance problem with the existing planners or have
> ideas to improve them feel free to file a JIRA.
>
> Best,
> Stamatis
>
> On Sun, Aug 2, 2020 at 10:38 PM Rui Wang <am...@apache.org> wrote:
>
> > I tried to dig into the Calcite codebase and found there is likely no way
> > to enable JDBC codepath:
> > 1. There is no planner config in CalciteConnectionProperty [1].
> > 2. In CalcitePrepareImpl, there is only one createPlannerFactory with
> one
> > createPlanner, in which VolcanoPlanner is used [2].
> >
> >
> > If my understanding is correct, then there could be an improvement that
> > adds a config to CalciteConnectionProperty, and use that config to
> control
> > which planner to use by JDBC approach.
> >
> >
> > [1]:
> >
> >
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/config/CalciteConnectionProperty.java#L37
> > [2]:
> >
> >
> https://github.com/apache/calcite/blob/2088488ac8327b19512a76a122cae2961fc551c3/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L427
> >
> >
> > -Rui
> >
> > On Sun, Aug 2, 2020 at 8:55 AM Meron Avigdor <
> meron.avigdor@gigaspaces.com
> > >
> > wrote:
> >
> > > Hi.
> > > I have implemented an extension of org.apache.calcite.jdbc.Driver, and
> I
> > > have the model handler extend org.apache.calcite.avatica.HandlerImpl
> > >
> > > For performance optimizations, I would like to test VolcanoPlanner vs.
> > > HepPlanner
> > > as the documentation says that the latter is faster.
> > >
> > > How is this to be configured? I could not find a hook point to return
> the
> > > planner to be used.
> > >
> > > Thank you.
> > > Meron
> > >
> >
>
Re: Avatica and HepPlanner
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hi Meron,
The two planners are not equivalent so the comparison is not totally fair.
Even if you apply the same ruleset it does not mean that you are going
to obtain the same plan.
The two planners are complementary to each other and they are used in
different stages of the optimization process thus it is not possible to use
exclusively one or the other. For the same reason I think it will be tricky
to
add such configuration.
If you notice a performance problem with the existing planners or have
ideas to improve them feel free to file a JIRA.
Best,
Stamatis
On Sun, Aug 2, 2020 at 10:38 PM Rui Wang <am...@apache.org> wrote:
> I tried to dig into the Calcite codebase and found there is likely no way
> to enable JDBC codepath:
> 1. There is no planner config in CalciteConnectionProperty [1].
> 2. In CalcitePrepareImpl, there is only one createPlannerFactory with one
> createPlanner, in which VolcanoPlanner is used [2].
>
>
> If my understanding is correct, then there could be an improvement that
> adds a config to CalciteConnectionProperty, and use that config to control
> which planner to use by JDBC approach.
>
>
> [1]:
>
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/config/CalciteConnectionProperty.java#L37
> [2]:
>
> https://github.com/apache/calcite/blob/2088488ac8327b19512a76a122cae2961fc551c3/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L427
>
>
> -Rui
>
> On Sun, Aug 2, 2020 at 8:55 AM Meron Avigdor <meron.avigdor@gigaspaces.com
> >
> wrote:
>
> > Hi.
> > I have implemented an extension of org.apache.calcite.jdbc.Driver, and I
> > have the model handler extend org.apache.calcite.avatica.HandlerImpl
> >
> > For performance optimizations, I would like to test VolcanoPlanner vs.
> > HepPlanner
> > as the documentation says that the latter is faster.
> >
> > How is this to be configured? I could not find a hook point to return the
> > planner to be used.
> >
> > Thank you.
> > Meron
> >
>
Re: Avatica and HepPlanner
Posted by Michael Mior <mm...@apache.org>.
I believe you're correct that it's not possible to select which
planner to use via JDBC. However, keep in mind that the HepPlanner and
VolcanoPlanner have different goals. VolcanoPlanner takes longer
because it is trying to minimize the expected cost of execution.
HepPlanner simply applies the given rules in order. (Which rules to
select for HepPlanner is another question as this itself is a
challenging problem.) The resulting query is not necessarily any
faster. Which planner is better will depend on the specific queries
being executed.
--
Michael Mior
mmior@apache.org
Le dim. 2 août 2020 à 15:38, Rui Wang <am...@apache.org> a écrit :
>
> I tried to dig into the Calcite codebase and found there is likely no way
> to enable JDBC codepath:
> 1. There is no planner config in CalciteConnectionProperty [1].
> 2. In CalcitePrepareImpl, there is only one createPlannerFactory with one
> createPlanner, in which VolcanoPlanner is used [2].
>
>
> If my understanding is correct, then there could be an improvement that
> adds a config to CalciteConnectionProperty, and use that config to control
> which planner to use by JDBC approach.
>
>
> [1]:
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/config/CalciteConnectionProperty.java#L37
> [2]:
> https://github.com/apache/calcite/blob/2088488ac8327b19512a76a122cae2961fc551c3/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L427
>
>
> -Rui
>
> On Sun, Aug 2, 2020 at 8:55 AM Meron Avigdor <me...@gigaspaces.com>
> wrote:
>
> > Hi.
> > I have implemented an extension of org.apache.calcite.jdbc.Driver, and I
> > have the model handler extend org.apache.calcite.avatica.HandlerImpl
> >
> > For performance optimizations, I would like to test VolcanoPlanner vs.
> > HepPlanner
> > as the documentation says that the latter is faster.
> >
> > How is this to be configured? I could not find a hook point to return the
> > planner to be used.
> >
> > Thank you.
> > Meron
> >
Re: Avatica and HepPlanner
Posted by Rui Wang <am...@apache.org>.
I tried to dig into the Calcite codebase and found there is likely no way
to enable JDBC codepath:
1. There is no planner config in CalciteConnectionProperty [1].
2. In CalcitePrepareImpl, there is only one createPlannerFactory with one
createPlanner, in which VolcanoPlanner is used [2].
If my understanding is correct, then there could be an improvement that
adds a config to CalciteConnectionProperty, and use that config to control
which planner to use by JDBC approach.
[1]:
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/config/CalciteConnectionProperty.java#L37
[2]:
https://github.com/apache/calcite/blob/2088488ac8327b19512a76a122cae2961fc551c3/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L427
-Rui
On Sun, Aug 2, 2020 at 8:55 AM Meron Avigdor <me...@gigaspaces.com>
wrote:
> Hi.
> I have implemented an extension of org.apache.calcite.jdbc.Driver, and I
> have the model handler extend org.apache.calcite.avatica.HandlerImpl
>
> For performance optimizations, I would like to test VolcanoPlanner vs.
> HepPlanner
> as the documentation says that the latter is faster.
>
> How is this to be configured? I could not find a hook point to return the
> planner to be used.
>
> Thank you.
> Meron
>