You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Taras Ledkov <tl...@gridgain.com> on 2021/05/26 11:26:31 UTC
Question: Hints & Rules
Hi,
I am trying to figure out how to use hints correctly.
My case:
I've registered hint for Aggregate node to force expand DISTINCT aggregate.
Because is some cases plan with DISTINCT aggregate to JOIN cannot be
chosen by the cost.
I see that different rules treat hints differently.
e.g.:
- AggregateExpandDistinctAggregatesRule - copy hints of the source node
for new aggregate nodes.
- AggregateReduceFunctionsRule - create new node and looses the original
hints.
Is is correct behavior and am I missing something at the hints/planner
logic?
--
Taras Ledkov
Mail-To: tledkov@gridgain.com
Re: Question: Hints & Rules
Posted by JiaTao Tao <ta...@gmail.com>.
Thanks Danny
Regards!
Aron Tao
Danny Chan <da...@apache.org> 于2021年5月27日周四 下午8:31写道：
> Jia Tao ~
> The hints is production ready, we have implements many hints internal for
> our Flink engine, such as the table hints and query hints.
>
> The `RelBuilder#project` was modified to support hints already, one reason
> for hints lost is that the projection are dropped for some reason(
> maybe some optimization), and you should fix it if you wanna keep that.
>
> Best,
> Danny Chan
>
> JiaTao Tao <ta...@gmail.com> 于2021年5月27日周四 下午4:22写道：
>
> > Hi
> > Indeed, there's some situation will lose hints, like RelBuilder#project,
> we
> > just construct a project without hints(ImmutableList.of()), so it will
> lose
> > hints too.
> > IMO, it's not ready for production, so finally, we give up this feature.
> > Currently, hints occurs some times in the mail list, proves that this is
> a
> > common usage, maybe we need to go over this feature again and make it
> > product ready.
> >
> >
> > Regards!
> >
> > Aron Tao
> >
> >
> > Danny Chan <da...@apache.org> 于2021年5月27日周四 上午11:01写道：
> >
> > > Hi ~
> > >
> > > We have interface RelHintsPropagator for hints propagation
> customization,
> > > but you may need to modify the code of the rule a little bit.
> > >
> > > Best,
> > > Danny Chan
> > >
> > > Taras Ledkov <tl...@gridgain.com> 于2021年5月26日周三 下午7:26写道：
> > >
> > > > Hi,
> > > >
> > > > I am trying to figure out how to use hints correctly.
> > > >
> > > > My case:
> > > > I've registered hint for Aggregate node to force expand DISTINCT
> > > aggregate.
> > > > Because is some cases plan with DISTINCT aggregate to JOIN cannot be
> > > > chosen by the cost.
> > > >
> > > > I see that different rules treat hints differently.
> > > > e.g.:
> > > > - AggregateExpandDistinctAggregatesRule - copy hints of the source
> node
> > > > for new aggregate nodes.
> > > > - AggregateReduceFunctionsRule - create new node and looses the
> > original
> > > > hints.
> > > >
> > > > Is is correct behavior and am I missing something at the
> hints/planner
> > > > logic?
> > > >
> > > > --
> > > > Taras Ledkov
> > > > Mail-To: tledkov@gridgain.com
> > > >
> > > >
> > >
> >
>
Re: Question: Hints & Rules
Posted by Danny Chan <da...@apache.org>.
Jia Tao ~
The hints is production ready, we have implements many hints internal for
our Flink engine, such as the table hints and query hints.
The `RelBuilder#project` was modified to support hints already, one reason
for hints lost is that the projection are dropped for some reason(
maybe some optimization), and you should fix it if you wanna keep that.
Best,
Danny Chan
JiaTao Tao <ta...@gmail.com> 于2021年5月27日周四 下午4:22写道：
> Hi
> Indeed, there's some situation will lose hints, like RelBuilder#project, we
> just construct a project without hints(ImmutableList.of()), so it will lose
> hints too.
> IMO, it's not ready for production, so finally, we give up this feature.
> Currently, hints occurs some times in the mail list, proves that this is a
> common usage, maybe we need to go over this feature again and make it
> product ready.
>
>
> Regards!
>
> Aron Tao
>
>
> Danny Chan <da...@apache.org> 于2021年5月27日周四 上午11:01写道：
>
> > Hi ~
> >
> > We have interface RelHintsPropagator for hints propagation customization,
> > but you may need to modify the code of the rule a little bit.
> >
> > Best,
> > Danny Chan
> >
> > Taras Ledkov <tl...@gridgain.com> 于2021年5月26日周三 下午7:26写道：
> >
> > > Hi,
> > >
> > > I am trying to figure out how to use hints correctly.
> > >
> > > My case:
> > > I've registered hint for Aggregate node to force expand DISTINCT
> > aggregate.
> > > Because is some cases plan with DISTINCT aggregate to JOIN cannot be
> > > chosen by the cost.
> > >
> > > I see that different rules treat hints differently.
> > > e.g.:
> > > - AggregateExpandDistinctAggregatesRule - copy hints of the source node
> > > for new aggregate nodes.
> > > - AggregateReduceFunctionsRule - create new node and looses the
> original
> > > hints.
> > >
> > > Is is correct behavior and am I missing something at the hints/planner
> > > logic?
> > >
> > > --
> > > Taras Ledkov
> > > Mail-To: tledkov@gridgain.com
> > >
> > >
> >
>
Re: Question: Hints & Rules
Posted by JiaTao Tao <ta...@gmail.com>.
Hi
Indeed, there's some situation will lose hints, like RelBuilder#project, we
just construct a project without hints(ImmutableList.of()), so it will lose
hints too.
IMO, it's not ready for production, so finally, we give up this feature.
Currently, hints occurs some times in the mail list, proves that this is a
common usage, maybe we need to go over this feature again and make it
product ready.
Regards!
Aron Tao
Danny Chan <da...@apache.org> 于2021年5月27日周四 上午11:01写道：
> Hi ~
>
> We have interface RelHintsPropagator for hints propagation customization,
> but you may need to modify the code of the rule a little bit.
>
> Best,
> Danny Chan
>
> Taras Ledkov <tl...@gridgain.com> 于2021年5月26日周三 下午7:26写道：
>
> > Hi,
> >
> > I am trying to figure out how to use hints correctly.
> >
> > My case:
> > I've registered hint for Aggregate node to force expand DISTINCT
> aggregate.
> > Because is some cases plan with DISTINCT aggregate to JOIN cannot be
> > chosen by the cost.
> >
> > I see that different rules treat hints differently.
> > e.g.:
> > - AggregateExpandDistinctAggregatesRule - copy hints of the source node
> > for new aggregate nodes.
> > - AggregateReduceFunctionsRule - create new node and looses the original
> > hints.
> >
> > Is is correct behavior and am I missing something at the hints/planner
> > logic?
> >
> > --
> > Taras Ledkov
> > Mail-To: tledkov@gridgain.com
> >
> >
>
Re: Question: Hints & Rules
Posted by Danny Chan <da...@apache.org>.
Hi ~
We have interface RelHintsPropagator for hints propagation customization,
but you may need to modify the code of the rule a little bit.
Best,
Danny Chan
Taras Ledkov <tl...@gridgain.com> 于2021年5月26日周三 下午7:26写道：
> Hi,
>
> I am trying to figure out how to use hints correctly.
>
> My case:
> I've registered hint for Aggregate node to force expand DISTINCT aggregate.
> Because is some cases plan with DISTINCT aggregate to JOIN cannot be
> chosen by the cost.
>
> I see that different rules treat hints differently.
> e.g.:
> - AggregateExpandDistinctAggregatesRule - copy hints of the source node
> for new aggregate nodes.
> - AggregateReduceFunctionsRule - create new node and looses the original
> hints.
>
> Is is correct behavior and am I missing something at the hints/planner
> logic?
>
> --
> Taras Ledkov
> Mail-To: tledkov@gridgain.com
>
>
Re: Question: Hints & Rules
Posted by Julian Hyde <jh...@apache.org>.
My fear, when we introduced hints, was that there would be many cases
where people would hope that rules propagate hints and the rules do
not; and that in some cases, the expected behavior isn't even clear.
Maybe this can be addressed by the power of open source: people find
an issue, submit a patch. But maybe not: people will discover that
most of the time, the hints they expected just aren't there. We'll
see. It depends on the proportion of people who find broken stuff and
fixing vs people who find broken stuff and walk away shaking their
head.
In this case, a pragmatic solution might be to convert the hint to a
query-level hint. Hints that apply to the whole query are not going to
be lost when rules are fired.
On Wed, May 26, 2021 at 2:05 PM Stamatis Zampetakis <za...@gmail.com> wrote:
>
> Hi Taras,
>
> I haven't used hints much but what you describe sounds like a bug.
>
> I would expect that rules should preserve hints if possible. I guess that
> if a rule removes an operator completely then it may not make sense to
> retain the hints.
>
> Best,
> Stamatis
>
> On Wed, May 26, 2021, 1:26 PM Taras Ledkov <tl...@gridgain.com> wrote:
>
> > Hi,
> >
> > I am trying to figure out how to use hints correctly.
> >
> > My case:
> > I've registered hint for Aggregate node to force expand DISTINCT aggregate.
> > Because is some cases plan with DISTINCT aggregate to JOIN cannot be
> > chosen by the cost.
> >
> > I see that different rules treat hints differently.
> > e.g.:
> > - AggregateExpandDistinctAggregatesRule - copy hints of the source node
> > for new aggregate nodes.
> > - AggregateReduceFunctionsRule - create new node and looses the original
> > hints.
> >
> > Is is correct behavior and am I missing something at the hints/planner
> > logic?
> >
> > --
> > Taras Ledkov
> > Mail-To: tledkov@gridgain.com
> >
> >
Re: Question: Hints & Rules
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hi Taras,
I haven't used hints much but what you describe sounds like a bug.
I would expect that rules should preserve hints if possible. I guess that
if a rule removes an operator completely then it may not make sense to
retain the hints.
Best,
Stamatis
On Wed, May 26, 2021, 1:26 PM Taras Ledkov <tl...@gridgain.com> wrote:
> Hi,
>
> I am trying to figure out how to use hints correctly.
>
> My case:
> I've registered hint for Aggregate node to force expand DISTINCT aggregate.
> Because is some cases plan with DISTINCT aggregate to JOIN cannot be
> chosen by the cost.
>
> I see that different rules treat hints differently.
> e.g.:
> - AggregateExpandDistinctAggregatesRule - copy hints of the source node
> for new aggregate nodes.
> - AggregateReduceFunctionsRule - create new node and looses the original
> hints.
>
> Is is correct behavior and am I missing something at the hints/planner
> logic?
>
> --
> Taras Ledkov
> Mail-To: tledkov@gridgain.com
>
>