You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Milinda Pathirage <mp...@umail.iu.edu> on 2015/08/30 02:36:50 UTC
What is the purpose of having Drill specific logical operator layer
and physical operator layer
Hi devs,
Does anybody know the reason behind having a Drill specific logical plan
layer (DrillRel) and a physical plan layer (Prel)?
Thanks in advance
Milinda
--
Milinda Pathirage
PhD Student | Research Assistant
School of Informatics and Computing | Data to Insight Center
Indiana University
twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org
Re: What is the purpose of having Drill specific logical operator
layer and physical operator layer
Posted by Julian Hyde <jh...@apache.org>.
...and then in very-physical planning you decide the the number of
nodes to use for your distributed hash join algorithm.
On Fri, Sep 4, 2015 at 11:47 AM, Hsuan Yi Chu <hy...@maprtech.com> wrote:
> Hi Milinda,
>
> For example, in Logical planning, Planner might decide using an INNER-JOIN
> is cheaper than using a FILTER on top a CARTESIAN-JOIN.
>
> Then later in Physical Planing, it further chooses an "implementation" of
> this JOIN, (e.g., HASH-JOIN v.s. MERGE-JOIN)
>
> On Fri, Sep 4, 2015 at 11:05 AM, Julian Hyde <jh...@apache.org> wrote:
>
>> Yes, Drill’s 3 layer architecture makes sense. The only thing I’d have
>> done differently is to put all 3 in the Calcite algebra, whereas Drill has
>> (I believe) a different physical algebra.
>>
>> > On Sep 4, 2015, at 11:02 AM, Milinda Pathirage <mp...@umail.iu.edu>
>> wrote:
>> >
>> > Thanks Ted.
>> >
>> > Milinda
>> >
>> > On Sun, Aug 30, 2015 at 12:52 AM, Ted Dunning <te...@gmail.com>
>> wrote:
>> >
>> >> Milinda,
>> >>
>> >> Factoring the version from SQL to an actual execution plan has a few
>> >> benefits. One of the biggest is that the logical plan doesn't vary if
>> the
>> >> cluster size or shape or locality changes, but the physical plan does.
>> >> This allows the SQL => logical translation and optimization to be
>> >> developed, tested and debugged independent of the cluster used.
>> >>
>> >>
>> >>
>> >> On Sat, Aug 29, 2015 at 5:36 PM, Milinda Pathirage <
>> mpathira@umail.iu.edu>
>> >> wrote:
>> >>
>> >>> Hi devs,
>> >>>
>> >>> Does anybody know the reason behind having a Drill specific logical
>> plan
>> >>> layer (DrillRel) and a physical plan layer (Prel)?
>> >>>
>> >>> Thanks in advance
>> >>> Milinda
>> >>>
>> >>>
>> >>> --
>> >>> Milinda Pathirage
>> >>>
>> >>> PhD Student | Research Assistant
>> >>> School of Informatics and Computing | Data to Insight Center
>> >>> Indiana University
>> >>>
>> >>> twitter: milindalakmal
>> >>> skype: milinda.pathirage
>> >>> blog: http://milinda.pathirage.org
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > Milinda Pathirage
>> >
>> > PhD Student | Research Assistant
>> > School of Informatics and Computing | Data to Insight Center
>> > Indiana University
>> >
>> > twitter: milindalakmal
>> > skype: milinda.pathirage
>> > blog: http://milinda.pathirage.org
>>
>>
Re: What is the purpose of having Drill specific logical operator
layer and physical operator layer
Posted by Hsuan Yi Chu <hy...@maprtech.com>.
Hi Milinda,
For example, in Logical planning, Planner might decide using an INNER-JOIN
is cheaper than using a FILTER on top a CARTESIAN-JOIN.
Then later in Physical Planing, it further chooses an "implementation" of
this JOIN, (e.g., HASH-JOIN v.s. MERGE-JOIN)
On Fri, Sep 4, 2015 at 11:05 AM, Julian Hyde <jh...@apache.org> wrote:
> Yes, Drill’s 3 layer architecture makes sense. The only thing I’d have
> done differently is to put all 3 in the Calcite algebra, whereas Drill has
> (I believe) a different physical algebra.
>
> > On Sep 4, 2015, at 11:02 AM, Milinda Pathirage <mp...@umail.iu.edu>
> wrote:
> >
> > Thanks Ted.
> >
> > Milinda
> >
> > On Sun, Aug 30, 2015 at 12:52 AM, Ted Dunning <te...@gmail.com>
> wrote:
> >
> >> Milinda,
> >>
> >> Factoring the version from SQL to an actual execution plan has a few
> >> benefits. One of the biggest is that the logical plan doesn't vary if
> the
> >> cluster size or shape or locality changes, but the physical plan does.
> >> This allows the SQL => logical translation and optimization to be
> >> developed, tested and debugged independent of the cluster used.
> >>
> >>
> >>
> >> On Sat, Aug 29, 2015 at 5:36 PM, Milinda Pathirage <
> mpathira@umail.iu.edu>
> >> wrote:
> >>
> >>> Hi devs,
> >>>
> >>> Does anybody know the reason behind having a Drill specific logical
> plan
> >>> layer (DrillRel) and a physical plan layer (Prel)?
> >>>
> >>> Thanks in advance
> >>> Milinda
> >>>
> >>>
> >>> --
> >>> Milinda Pathirage
> >>>
> >>> PhD Student | Research Assistant
> >>> School of Informatics and Computing | Data to Insight Center
> >>> Indiana University
> >>>
> >>> twitter: milindalakmal
> >>> skype: milinda.pathirage
> >>> blog: http://milinda.pathirage.org
> >>>
> >>
> >
> >
> >
> > --
> > Milinda Pathirage
> >
> > PhD Student | Research Assistant
> > School of Informatics and Computing | Data to Insight Center
> > Indiana University
> >
> > twitter: milindalakmal
> > skype: milinda.pathirage
> > blog: http://milinda.pathirage.org
>
>
Re: What is the purpose of having Drill specific logical operator layer and physical operator layer
Posted by Julian Hyde <jh...@apache.org>.
Yes, Drill’s 3 layer architecture makes sense. The only thing I’d have done differently is to put all 3 in the Calcite algebra, whereas Drill has (I believe) a different physical algebra.
> On Sep 4, 2015, at 11:02 AM, Milinda Pathirage <mp...@umail.iu.edu> wrote:
>
> Thanks Ted.
>
> Milinda
>
> On Sun, Aug 30, 2015 at 12:52 AM, Ted Dunning <te...@gmail.com> wrote:
>
>> Milinda,
>>
>> Factoring the version from SQL to an actual execution plan has a few
>> benefits. One of the biggest is that the logical plan doesn't vary if the
>> cluster size or shape or locality changes, but the physical plan does.
>> This allows the SQL => logical translation and optimization to be
>> developed, tested and debugged independent of the cluster used.
>>
>>
>>
>> On Sat, Aug 29, 2015 at 5:36 PM, Milinda Pathirage <mp...@umail.iu.edu>
>> wrote:
>>
>>> Hi devs,
>>>
>>> Does anybody know the reason behind having a Drill specific logical plan
>>> layer (DrillRel) and a physical plan layer (Prel)?
>>>
>>> Thanks in advance
>>> Milinda
>>>
>>>
>>> --
>>> Milinda Pathirage
>>>
>>> PhD Student | Research Assistant
>>> School of Informatics and Computing | Data to Insight Center
>>> Indiana University
>>>
>>> twitter: milindalakmal
>>> skype: milinda.pathirage
>>> blog: http://milinda.pathirage.org
>>>
>>
>
>
>
> --
> Milinda Pathirage
>
> PhD Student | Research Assistant
> School of Informatics and Computing | Data to Insight Center
> Indiana University
>
> twitter: milindalakmal
> skype: milinda.pathirage
> blog: http://milinda.pathirage.org
Re: What is the purpose of having Drill specific logical operator
layer and physical operator layer
Posted by Milinda Pathirage <mp...@umail.iu.edu>.
Thanks Ted.
Milinda
On Sun, Aug 30, 2015 at 12:52 AM, Ted Dunning <te...@gmail.com> wrote:
> Milinda,
>
> Factoring the version from SQL to an actual execution plan has a few
> benefits. One of the biggest is that the logical plan doesn't vary if the
> cluster size or shape or locality changes, but the physical plan does.
> This allows the SQL => logical translation and optimization to be
> developed, tested and debugged independent of the cluster used.
>
>
>
> On Sat, Aug 29, 2015 at 5:36 PM, Milinda Pathirage <mp...@umail.iu.edu>
> wrote:
>
> > Hi devs,
> >
> > Does anybody know the reason behind having a Drill specific logical plan
> > layer (DrillRel) and a physical plan layer (Prel)?
> >
> > Thanks in advance
> > Milinda
> >
> >
> > --
> > Milinda Pathirage
> >
> > PhD Student | Research Assistant
> > School of Informatics and Computing | Data to Insight Center
> > Indiana University
> >
> > twitter: milindalakmal
> > skype: milinda.pathirage
> > blog: http://milinda.pathirage.org
> >
>
--
Milinda Pathirage
PhD Student | Research Assistant
School of Informatics and Computing | Data to Insight Center
Indiana University
twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org
Re: What is the purpose of having Drill specific logical operator
layer and physical operator layer
Posted by Ted Dunning <te...@gmail.com>.
Milinda,
Factoring the version from SQL to an actual execution plan has a few
benefits. One of the biggest is that the logical plan doesn't vary if the
cluster size or shape or locality changes, but the physical plan does.
This allows the SQL => logical translation and optimization to be
developed, tested and debugged independent of the cluster used.
On Sat, Aug 29, 2015 at 5:36 PM, Milinda Pathirage <mp...@umail.iu.edu>
wrote:
> Hi devs,
>
> Does anybody know the reason behind having a Drill specific logical plan
> layer (DrillRel) and a physical plan layer (Prel)?
>
> Thanks in advance
> Milinda
>
>
> --
> Milinda Pathirage
>
> PhD Student | Research Assistant
> School of Informatics and Computing | Data to Insight Center
> Indiana University
>
> twitter: milindalakmal
> skype: milinda.pathirage
> blog: http://milinda.pathirage.org
>