You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Milinda Pathirage <mp...@umail.iu.edu> on 2015/03/18 15:51:18 UTC
Merging Project and Scan when there are RexCalls in projections
Hi Julian,
I am trying to understand why Project and Scan merging is avoided when
there are projections which are RexCalls. Is this a purely implementation
related rule (such as separation of concerns or keeping scan as simple as
possible) or is there any theoretical background for this.
I am asking this because I discovered that Project#getMapping returns null
when there are RexCalls in the projection which prevent ProjectableScanRule
from transforming the logical expression tree.
Thanks in advance,
Milinda
--
Milinda Pathirage
PhD Student | Research Assistant
School of Informatics and Computing | Data to Insight Center
Indiana University
twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org
Re: Merging Project and Scan when there are RexCalls in projections
Posted by Milinda Pathirage <mp...@umail.iu.edu>.
Hi Julian,
Thanks for the response.
Yes indeed, it looks like there will be several Samza specific push-downs
and RelNodes. I'm still trying to understand how conventions work and what
is there for Samza. But I'm sure we will need a Samza specific convention
as we grow.
Milinda
On Wed, Mar 18, 2015 at 4:53 PM, Julian Hyde <ju...@hydromatic.net> wrote:
> ProjectableTable and ProjectableFilterableTable are compromises. They make
> it easy for someone to write a Calcite adapter for a data source, and
> perform the most common push-downs. 80% of the value for 20% of the effort.
>
> I thought it was worth pushing down filter expressions (e.g. x > 5 AND y
> is null) but I didn’t think it was worth pushing down projections. The
> ProjectableFilterableTable.scan API reflects this.
>
> If someone is an expert in Calcite, and has a data source that is capable
> of doing significant computation (e.g. covers a lot of the built-in SQL
> functions, user-defined functions, aggregation…) then they should build a
> “real” adapter. They will also need to write their own RelNodes, push-down
> rules, and probably their own convention. FWIW I think Samza is in that
> category.
>
> Julian
>
>
> On Mar 18, 2015, at 7:51 AM, Milinda Pathirage <mp...@umail.iu.edu>
> wrote:
>
> > Hi Julian,
> >
> > I am trying to understand why Project and Scan merging is avoided when
> > there are projections which are RexCalls. Is this a purely implementation
> > related rule (such as separation of concerns or keeping scan as simple as
> > possible) or is there any theoretical background for this.
> >
> > I am asking this because I discovered that Project#getMapping returns
> null
> > when there are RexCalls in the projection which prevent
> ProjectableScanRule
> > from transforming the logical expression tree.
> >
> > Thanks in advance,
> > Milinda
> >
> > --
> > Milinda Pathirage
> >
> > PhD Student | Research Assistant
> > School of Informatics and Computing | Data to Insight Center
> > Indiana University
> >
> > twitter: milindalakmal
> > skype: milinda.pathirage
> > blog: http://milinda.pathirage.org
>
>
--
Milinda Pathirage
PhD Student | Research Assistant
School of Informatics and Computing | Data to Insight Center
Indiana University
twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org
Re: Merging Project and Scan when there are RexCalls in projections
Posted by Julian Hyde <ju...@hydromatic.net>.
ProjectableTable and ProjectableFilterableTable are compromises. They make it easy for someone to write a Calcite adapter for a data source, and perform the most common push-downs. 80% of the value for 20% of the effort.
I thought it was worth pushing down filter expressions (e.g. x > 5 AND y is null) but I didn’t think it was worth pushing down projections. The ProjectableFilterableTable.scan API reflects this.
If someone is an expert in Calcite, and has a data source that is capable of doing significant computation (e.g. covers a lot of the built-in SQL functions, user-defined functions, aggregation…) then they should build a “real” adapter. They will also need to write their own RelNodes, push-down rules, and probably their own convention. FWIW I think Samza is in that category.
Julian
On Mar 18, 2015, at 7:51 AM, Milinda Pathirage <mp...@umail.iu.edu> wrote:
> Hi Julian,
>
> I am trying to understand why Project and Scan merging is avoided when
> there are projections which are RexCalls. Is this a purely implementation
> related rule (such as separation of concerns or keeping scan as simple as
> possible) or is there any theoretical background for this.
>
> I am asking this because I discovered that Project#getMapping returns null
> when there are RexCalls in the projection which prevent ProjectableScanRule
> from transforming the logical expression tree.
>
> Thanks in advance,
> Milinda
>
> --
> Milinda Pathirage
>
> PhD Student | Research Assistant
> School of Informatics and Computing | Data to Insight Center
> Indiana University
>
> twitter: milindalakmal
> skype: milinda.pathirage
> blog: http://milinda.pathirage.org