You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Milinda Pathirage <mp...@umail.iu.edu> on 2015/09/04 20:02:44 UTC
Re: What is the purpose of having Drill specific logical operator
layer and physical operator layer
Thanks Ted.
Milinda
On Sun, Aug 30, 2015 at 12:52 AM, Ted Dunning <te...@gmail.com> wrote:
> Milinda,
>
> Factoring the version from SQL to an actual execution plan has a few
> benefits. One of the biggest is that the logical plan doesn't vary if the
> cluster size or shape or locality changes, but the physical plan does.
> This allows the SQL => logical translation and optimization to be
> developed, tested and debugged independent of the cluster used.
>
>
>
> On Sat, Aug 29, 2015 at 5:36 PM, Milinda Pathirage <mp...@umail.iu.edu>
> wrote:
>
> > Hi devs,
> >
> > Does anybody know the reason behind having a Drill specific logical plan
> > layer (DrillRel) and a physical plan layer (Prel)?
> >
> > Thanks in advance
> > Milinda
> >
> >
> > --
> > Milinda Pathirage
> >
> > PhD Student | Research Assistant
> > School of Informatics and Computing | Data to Insight Center
> > Indiana University
> >
> > twitter: milindalakmal
> > skype: milinda.pathirage
> > blog: http://milinda.pathirage.org
> >
>
--
Milinda Pathirage
PhD Student | Research Assistant
School of Informatics and Computing | Data to Insight Center
Indiana University
twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org
Re: What is the purpose of having Drill specific logical operator
layer and physical operator layer
Posted by Julian Hyde <jh...@apache.org>.
...and then in very-physical planning you decide the the number of
nodes to use for your distributed hash join algorithm.
On Fri, Sep 4, 2015 at 11:47 AM, Hsuan Yi Chu <hy...@maprtech.com> wrote:
> Hi Milinda,
>
> For example, in Logical planning, Planner might decide using an INNER-JOIN
> is cheaper than using a FILTER on top a CARTESIAN-JOIN.
>
> Then later in Physical Planing, it further chooses an "implementation" of
> this JOIN, (e.g., HASH-JOIN v.s. MERGE-JOIN)
>
> On Fri, Sep 4, 2015 at 11:05 AM, Julian Hyde <jh...@apache.org> wrote:
>
>> Yes, Drill’s 3 layer architecture makes sense. The only thing I’d have
>> done differently is to put all 3 in the Calcite algebra, whereas Drill has
>> (I believe) a different physical algebra.
>>
>> > On Sep 4, 2015, at 11:02 AM, Milinda Pathirage <mp...@umail.iu.edu>
>> wrote:
>> >
>> > Thanks Ted.
>> >
>> > Milinda
>> >
>> > On Sun, Aug 30, 2015 at 12:52 AM, Ted Dunning <te...@gmail.com>
>> wrote:
>> >
>> >> Milinda,
>> >>
>> >> Factoring the version from SQL to an actual execution plan has a few
>> >> benefits. One of the biggest is that the logical plan doesn't vary if
>> the
>> >> cluster size or shape or locality changes, but the physical plan does.
>> >> This allows the SQL => logical translation and optimization to be
>> >> developed, tested and debugged independent of the cluster used.
>> >>
>> >>
>> >>
>> >> On Sat, Aug 29, 2015 at 5:36 PM, Milinda Pathirage <
>> mpathira@umail.iu.edu>
>> >> wrote:
>> >>
>> >>> Hi devs,
>> >>>
>> >>> Does anybody know the reason behind having a Drill specific logical
>> plan
>> >>> layer (DrillRel) and a physical plan layer (Prel)?
>> >>>
>> >>> Thanks in advance
>> >>> Milinda
>> >>>
>> >>>
>> >>> --
>> >>> Milinda Pathirage
>> >>>
>> >>> PhD Student | Research Assistant
>> >>> School of Informatics and Computing | Data to Insight Center
>> >>> Indiana University
>> >>>
>> >>> twitter: milindalakmal
>> >>> skype: milinda.pathirage
>> >>> blog: http://milinda.pathirage.org
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > Milinda Pathirage
>> >
>> > PhD Student | Research Assistant
>> > School of Informatics and Computing | Data to Insight Center
>> > Indiana University
>> >
>> > twitter: milindalakmal
>> > skype: milinda.pathirage
>> > blog: http://milinda.pathirage.org
>>
>>
Re: What is the purpose of having Drill specific logical operator
layer and physical operator layer
Posted by Hsuan Yi Chu <hy...@maprtech.com>.
Hi Milinda,
For example, in Logical planning, Planner might decide using an INNER-JOIN
is cheaper than using a FILTER on top a CARTESIAN-JOIN.
Then later in Physical Planing, it further chooses an "implementation" of
this JOIN, (e.g., HASH-JOIN v.s. MERGE-JOIN)
On Fri, Sep 4, 2015 at 11:05 AM, Julian Hyde <jh...@apache.org> wrote:
> Yes, Drill’s 3 layer architecture makes sense. The only thing I’d have
> done differently is to put all 3 in the Calcite algebra, whereas Drill has
> (I believe) a different physical algebra.
>
> > On Sep 4, 2015, at 11:02 AM, Milinda Pathirage <mp...@umail.iu.edu>
> wrote:
> >
> > Thanks Ted.
> >
> > Milinda
> >
> > On Sun, Aug 30, 2015 at 12:52 AM, Ted Dunning <te...@gmail.com>
> wrote:
> >
> >> Milinda,
> >>
> >> Factoring the version from SQL to an actual execution plan has a few
> >> benefits. One of the biggest is that the logical plan doesn't vary if
> the
> >> cluster size or shape or locality changes, but the physical plan does.
> >> This allows the SQL => logical translation and optimization to be
> >> developed, tested and debugged independent of the cluster used.
> >>
> >>
> >>
> >> On Sat, Aug 29, 2015 at 5:36 PM, Milinda Pathirage <
> mpathira@umail.iu.edu>
> >> wrote:
> >>
> >>> Hi devs,
> >>>
> >>> Does anybody know the reason behind having a Drill specific logical
> plan
> >>> layer (DrillRel) and a physical plan layer (Prel)?
> >>>
> >>> Thanks in advance
> >>> Milinda
> >>>
> >>>
> >>> --
> >>> Milinda Pathirage
> >>>
> >>> PhD Student | Research Assistant
> >>> School of Informatics and Computing | Data to Insight Center
> >>> Indiana University
> >>>
> >>> twitter: milindalakmal
> >>> skype: milinda.pathirage
> >>> blog: http://milinda.pathirage.org
> >>>
> >>
> >
> >
> >
> > --
> > Milinda Pathirage
> >
> > PhD Student | Research Assistant
> > School of Informatics and Computing | Data to Insight Center
> > Indiana University
> >
> > twitter: milindalakmal
> > skype: milinda.pathirage
> > blog: http://milinda.pathirage.org
>
>
Re: What is the purpose of having Drill specific logical operator layer and physical operator layer
Posted by Julian Hyde <jh...@apache.org>.
Yes, Drill’s 3 layer architecture makes sense. The only thing I’d have done differently is to put all 3 in the Calcite algebra, whereas Drill has (I believe) a different physical algebra.
> On Sep 4, 2015, at 11:02 AM, Milinda Pathirage <mp...@umail.iu.edu> wrote:
>
> Thanks Ted.
>
> Milinda
>
> On Sun, Aug 30, 2015 at 12:52 AM, Ted Dunning <te...@gmail.com> wrote:
>
>> Milinda,
>>
>> Factoring the version from SQL to an actual execution plan has a few
>> benefits. One of the biggest is that the logical plan doesn't vary if the
>> cluster size or shape or locality changes, but the physical plan does.
>> This allows the SQL => logical translation and optimization to be
>> developed, tested and debugged independent of the cluster used.
>>
>>
>>
>> On Sat, Aug 29, 2015 at 5:36 PM, Milinda Pathirage <mp...@umail.iu.edu>
>> wrote:
>>
>>> Hi devs,
>>>
>>> Does anybody know the reason behind having a Drill specific logical plan
>>> layer (DrillRel) and a physical plan layer (Prel)?
>>>
>>> Thanks in advance
>>> Milinda
>>>
>>>
>>> --
>>> Milinda Pathirage
>>>
>>> PhD Student | Research Assistant
>>> School of Informatics and Computing | Data to Insight Center
>>> Indiana University
>>>
>>> twitter: milindalakmal
>>> skype: milinda.pathirage
>>> blog: http://milinda.pathirage.org
>>>
>>
>
>
>
> --
> Milinda Pathirage
>
> PhD Student | Research Assistant
> School of Informatics and Computing | Data to Insight Center
> Indiana University
>
> twitter: milindalakmal
> skype: milinda.pathirage
> blog: http://milinda.pathirage.org