You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Alexander Reshetov <al...@gmail.com> on 2015/11/03 17:09:02 UTC
Re: Is it possible to make logical query rewrite?
Thanks Maryann.
Trying to debug what's going on I created ArchiveConverterRule (and
registered this rule).
Rule implements ConverterRule and add convert(), which returns
ArchiveEnumerableSort.
So basically ArchiveEnumerableSort should be my replacement of EnumerableSort.
But when I call explain plan for select I get EnumerableSort anyway,
while I expect to see
implemented ArchiveEnumerableSort.
Should it work as I expect, or there is something other that I need to do?
On Sat, Oct 31, 2015 at 3:26 AM, Maryann Xue <ma...@gmail.com> wrote:
> Oh, it's not on master branch yet. It's the "calcite" branch, and for
> PhoenixTable specifically, you can go:
> https://git1-us-west.apache.org/repos/asf?p=phoenix.git;a=blob;f=phoenix-core/src/main/java/org/apache/phoenix/calcite/PhoenixTable.java;h=4be745036335443bc1bcaae07acf1f32fda6c59a;hb=refs/heads/calcite
> .
>
> On Fri, Oct 30, 2015 at 4:36 PM, Alexander Reshetov <
> alexander.v.reshetov@gmail.com> wrote:
>
>> Could you please point me to PhoenixTable in repo? Unfortunately I
>> could not find it.
>>
>> On Fri, Oct 30, 2015 at 10:33 PM, Maryann Xue <ma...@gmail.com>
>> wrote:
>> > SortRemoveRule.INSTANCE is already in the default rule set. I think what
>> > you should do is just make your table return the right collation trait.
>> For
>> > example in Phoenix we have a Table implementation called PhoenixTable
>> which
>> > overrides getStatistic() and implements getCollations() of Statistic.
>> >
>> > On Fri, Oct 30, 2015 at 1:40 PM, Alexander Reshetov <
>> > alexander.v.reshetov@gmail.com> wrote:
>> >
>> >> Is it correct that I need to override
>> >> org.apache.calcite.rel.AbstractRelNode.register and add rule
>> >> SortRemoveRule.INSTANCE ?
>> >>
>> >> But while it checks RelCollation, then I should also somehow set it.
>> >> But I could not find how to do it.
>> >>
>> >> On Fri, Oct 30, 2015 at 4:23 PM, Alexander Reshetov
>> >> <al...@gmail.com> wrote:
>> >> > Here some clarification of my intends.
>> >> >
>> >> > Calcite choosing next plan:
>> >> >
>> >> > +------+
>> >> > | PLAN |
>> >> > +------+
>> >> > | EnumerableSort(sort0=[$0], dir0=[DESC])
>> >> > ArchiveTableScan(table=[[MAIN, SUB]], fields=[[0, 1, 2, 3, 4, 5, 6,
>> >> > 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]])
>> >> > |
>> >> > +------+
>> >> >
>> >> >
>> >> > The problem is invocation of EnumerableSort. Because data already
>> >> > sorted by one column. So I need to force exclusion of EnumerableSort
>> >> > from planner.
>> >> >
>> >> > Do I need implement some interface or override some method to achieve
>> >> this?
>> >> >
>> >> > On Fri, Oct 30, 2015 at 9:51 AM, Julian Hyde <jh...@apache.org>
>> wrote:
>> >> >> Yes, this is possible. In fact Phoenix does exactly this.
>> >> >>
>> >> >> Take a look at https://issues.apache.org/jira/browse/CALCITE-818 and
>> >> https://issues.apache.org/jira/browse/CALCITE-71, which were fixed
>> along
>> >> the way.
>> >> >>
>> >> >>> On Oct 29, 2015, at 1:16 AM, Alexander Reshetov <
>> >> alexander.v.reshetov@gmail.com> wrote:
>> >> >>>
>> >> >>> Hello,
>> >> >>>
>> >> >>> I have kinda theoretical question. But it should help to understand
>> >> >>> overall Calcite's possibility.
>> >> >>>
>> >> >>> Suppose there is some abstract format/table, which supports only
>> >> >>> sorting by one column.
>> >> >>>
>> >> >>> Let's say there is next query passed to calcite:
>> >> >>>
>> >> >>> select * from table order by id
>> >> >>>
>> >> >>> Is it possible to rewrite query tree so it will change this request
>> >> into this?
>> >> >>>
>> >> >>> select * from table
>> >> >>>
>> >> >>> In this way after logical query rewrite Calcite should not be
>> >> >>> concerned about sorting at all.
>> >> >>
>> >>
>>
Re: Is it possible to make logical query rewrite?
Posted by Alexander Reshetov <al...@gmail.com>.
Thanks, Maryann.
Yes, offset and fetch was set to null.
Finally I achieved EnumerableSort rewriting with my implementation
(ArchiveEnumerableSort) by unregistering EnumerableSort from planner
in TableScan implementation (while keeping registration of my
converter rule).
planner.removeRule(EnumerableRules.ENUMERABLE_SORT_RULE);
After this planner chose as expected ArchiveEnumerableSort.
Seems that Calcite prefer EnumerableRules despite if any other presented.
And in my implementation I was able to remove sorting of column, which is
already sorted in my dataset.
On Wed, Nov 4, 2015 at 6:29 AM, Maryann Xue <ma...@gmail.com> wrote:
> Hi Alexander,
>
> Not sure if it's the problem you have, but it might be worth checking if
> your ArchiveEnumerableSort contains "collation" only, i.e. offset==null &&
> fetch==null. The SortRemoveRule deals with sort-only Sort rels, if either
> offset or fetch is not null this rule won't apply. Let me know if it helps.
>
>
> Thanks,
> Maryann
>
> On Tue, Nov 3, 2015 at 11:09 AM, Alexander Reshetov <
> alexander.v.reshetov@gmail.com> wrote:
>
>> Thanks Maryann.
>>
>> Trying to debug what's going on I created ArchiveConverterRule (and
>> registered this rule).
>> Rule implements ConverterRule and add convert(), which returns
>> ArchiveEnumerableSort.
>>
>> So basically ArchiveEnumerableSort should be my replacement of
>> EnumerableSort.
>> But when I call explain plan for select I get EnumerableSort anyway,
>> while I expect to see
>> implemented ArchiveEnumerableSort.
>>
>> Should it work as I expect, or there is something other that I need to do?
>>
>> On Sat, Oct 31, 2015 at 3:26 AM, Maryann Xue <ma...@gmail.com>
>> wrote:
>> > Oh, it's not on master branch yet. It's the "calcite" branch, and for
>> > PhoenixTable specifically, you can go:
>> >
>> https://git1-us-west.apache.org/repos/asf?p=phoenix.git;a=blob;f=phoenix-core/src/main/java/org/apache/phoenix/calcite/PhoenixTable.java;h=4be745036335443bc1bcaae07acf1f32fda6c59a;hb=refs/heads/calcite
>> > .
>> >
>> > On Fri, Oct 30, 2015 at 4:36 PM, Alexander Reshetov <
>> > alexander.v.reshetov@gmail.com> wrote:
>> >
>> >> Could you please point me to PhoenixTable in repo? Unfortunately I
>> >> could not find it.
>> >>
>> >> On Fri, Oct 30, 2015 at 10:33 PM, Maryann Xue <ma...@gmail.com>
>> >> wrote:
>> >> > SortRemoveRule.INSTANCE is already in the default rule set. I think
>> what
>> >> > you should do is just make your table return the right collation
>> trait.
>> >> For
>> >> > example in Phoenix we have a Table implementation called PhoenixTable
>> >> which
>> >> > overrides getStatistic() and implements getCollations() of Statistic.
>> >> >
>> >> > On Fri, Oct 30, 2015 at 1:40 PM, Alexander Reshetov <
>> >> > alexander.v.reshetov@gmail.com> wrote:
>> >> >
>> >> >> Is it correct that I need to override
>> >> >> org.apache.calcite.rel.AbstractRelNode.register and add rule
>> >> >> SortRemoveRule.INSTANCE ?
>> >> >>
>> >> >> But while it checks RelCollation, then I should also somehow set it.
>> >> >> But I could not find how to do it.
>> >> >>
>> >> >> On Fri, Oct 30, 2015 at 4:23 PM, Alexander Reshetov
>> >> >> <al...@gmail.com> wrote:
>> >> >> > Here some clarification of my intends.
>> >> >> >
>> >> >> > Calcite choosing next plan:
>> >> >> >
>> >> >> > +------+
>> >> >> > | PLAN |
>> >> >> > +------+
>> >> >> > | EnumerableSort(sort0=[$0], dir0=[DESC])
>> >> >> > ArchiveTableScan(table=[[MAIN, SUB]], fields=[[0, 1, 2, 3, 4, 5,
>> 6,
>> >> >> > 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]])
>> >> >> > |
>> >> >> > +------+
>> >> >> >
>> >> >> >
>> >> >> > The problem is invocation of EnumerableSort. Because data already
>> >> >> > sorted by one column. So I need to force exclusion of
>> EnumerableSort
>> >> >> > from planner.
>> >> >> >
>> >> >> > Do I need implement some interface or override some method to
>> achieve
>> >> >> this?
>> >> >> >
>> >> >> > On Fri, Oct 30, 2015 at 9:51 AM, Julian Hyde <jh...@apache.org>
>> >> wrote:
>> >> >> >> Yes, this is possible. In fact Phoenix does exactly this.
>> >> >> >>
>> >> >> >> Take a look at https://issues.apache.org/jira/browse/CALCITE-818
>> and
>> >> >> https://issues.apache.org/jira/browse/CALCITE-71, which were fixed
>> >> along
>> >> >> the way.
>> >> >> >>
>> >> >> >>> On Oct 29, 2015, at 1:16 AM, Alexander Reshetov <
>> >> >> alexander.v.reshetov@gmail.com> wrote:
>> >> >> >>>
>> >> >> >>> Hello,
>> >> >> >>>
>> >> >> >>> I have kinda theoretical question. But it should help to
>> understand
>> >> >> >>> overall Calcite's possibility.
>> >> >> >>>
>> >> >> >>> Suppose there is some abstract format/table, which supports only
>> >> >> >>> sorting by one column.
>> >> >> >>>
>> >> >> >>> Let's say there is next query passed to calcite:
>> >> >> >>>
>> >> >> >>> select * from table order by id
>> >> >> >>>
>> >> >> >>> Is it possible to rewrite query tree so it will change this
>> request
>> >> >> into this?
>> >> >> >>>
>> >> >> >>> select * from table
>> >> >> >>>
>> >> >> >>> In this way after logical query rewrite Calcite should not be
>> >> >> >>> concerned about sorting at all.
>> >> >> >>
>> >> >>
>> >>
>>
Re: Is it possible to make logical query rewrite?
Posted by Maryann Xue <ma...@gmail.com>.
Hi Alexander,
Not sure if it's the problem you have, but it might be worth checking if
your ArchiveEnumerableSort contains "collation" only, i.e. offset==null &&
fetch==null. The SortRemoveRule deals with sort-only Sort rels, if either
offset or fetch is not null this rule won't apply. Let me know if it helps.
Thanks,
Maryann
On Tue, Nov 3, 2015 at 11:09 AM, Alexander Reshetov <
alexander.v.reshetov@gmail.com> wrote:
> Thanks Maryann.
>
> Trying to debug what's going on I created ArchiveConverterRule (and
> registered this rule).
> Rule implements ConverterRule and add convert(), which returns
> ArchiveEnumerableSort.
>
> So basically ArchiveEnumerableSort should be my replacement of
> EnumerableSort.
> But when I call explain plan for select I get EnumerableSort anyway,
> while I expect to see
> implemented ArchiveEnumerableSort.
>
> Should it work as I expect, or there is something other that I need to do?
>
> On Sat, Oct 31, 2015 at 3:26 AM, Maryann Xue <ma...@gmail.com>
> wrote:
> > Oh, it's not on master branch yet. It's the "calcite" branch, and for
> > PhoenixTable specifically, you can go:
> >
> https://git1-us-west.apache.org/repos/asf?p=phoenix.git;a=blob;f=phoenix-core/src/main/java/org/apache/phoenix/calcite/PhoenixTable.java;h=4be745036335443bc1bcaae07acf1f32fda6c59a;hb=refs/heads/calcite
> > .
> >
> > On Fri, Oct 30, 2015 at 4:36 PM, Alexander Reshetov <
> > alexander.v.reshetov@gmail.com> wrote:
> >
> >> Could you please point me to PhoenixTable in repo? Unfortunately I
> >> could not find it.
> >>
> >> On Fri, Oct 30, 2015 at 10:33 PM, Maryann Xue <ma...@gmail.com>
> >> wrote:
> >> > SortRemoveRule.INSTANCE is already in the default rule set. I think
> what
> >> > you should do is just make your table return the right collation
> trait.
> >> For
> >> > example in Phoenix we have a Table implementation called PhoenixTable
> >> which
> >> > overrides getStatistic() and implements getCollations() of Statistic.
> >> >
> >> > On Fri, Oct 30, 2015 at 1:40 PM, Alexander Reshetov <
> >> > alexander.v.reshetov@gmail.com> wrote:
> >> >
> >> >> Is it correct that I need to override
> >> >> org.apache.calcite.rel.AbstractRelNode.register and add rule
> >> >> SortRemoveRule.INSTANCE ?
> >> >>
> >> >> But while it checks RelCollation, then I should also somehow set it.
> >> >> But I could not find how to do it.
> >> >>
> >> >> On Fri, Oct 30, 2015 at 4:23 PM, Alexander Reshetov
> >> >> <al...@gmail.com> wrote:
> >> >> > Here some clarification of my intends.
> >> >> >
> >> >> > Calcite choosing next plan:
> >> >> >
> >> >> > +------+
> >> >> > | PLAN |
> >> >> > +------+
> >> >> > | EnumerableSort(sort0=[$0], dir0=[DESC])
> >> >> > ArchiveTableScan(table=[[MAIN, SUB]], fields=[[0, 1, 2, 3, 4, 5,
> 6,
> >> >> > 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]])
> >> >> > |
> >> >> > +------+
> >> >> >
> >> >> >
> >> >> > The problem is invocation of EnumerableSort. Because data already
> >> >> > sorted by one column. So I need to force exclusion of
> EnumerableSort
> >> >> > from planner.
> >> >> >
> >> >> > Do I need implement some interface or override some method to
> achieve
> >> >> this?
> >> >> >
> >> >> > On Fri, Oct 30, 2015 at 9:51 AM, Julian Hyde <jh...@apache.org>
> >> wrote:
> >> >> >> Yes, this is possible. In fact Phoenix does exactly this.
> >> >> >>
> >> >> >> Take a look at https://issues.apache.org/jira/browse/CALCITE-818
> and
> >> >> https://issues.apache.org/jira/browse/CALCITE-71, which were fixed
> >> along
> >> >> the way.
> >> >> >>
> >> >> >>> On Oct 29, 2015, at 1:16 AM, Alexander Reshetov <
> >> >> alexander.v.reshetov@gmail.com> wrote:
> >> >> >>>
> >> >> >>> Hello,
> >> >> >>>
> >> >> >>> I have kinda theoretical question. But it should help to
> understand
> >> >> >>> overall Calcite's possibility.
> >> >> >>>
> >> >> >>> Suppose there is some abstract format/table, which supports only
> >> >> >>> sorting by one column.
> >> >> >>>
> >> >> >>> Let's say there is next query passed to calcite:
> >> >> >>>
> >> >> >>> select * from table order by id
> >> >> >>>
> >> >> >>> Is it possible to rewrite query tree so it will change this
> request
> >> >> into this?
> >> >> >>>
> >> >> >>> select * from table
> >> >> >>>
> >> >> >>> In this way after logical query rewrite Calcite should not be
> >> >> >>> concerned about sorting at all.
> >> >> >>
> >> >>
> >>
>