You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Renato Marroquín Mogrovejo <re...@gmail.com> on 2010/07/02 02:24:28 UTC
Re: Bug in new logical optimizer framework?
Hi,
I am also interested in this logical plan optimization framework
functionality. You mentioned that rules are being developed, could you
explain a little bit more about them? Are they like the classic logical
optimizations (early projection, early filtering, among others)?
Thanks in advance.
Renato M.
logical plan optimization framework
2010/6/28 Alan Gates <ga...@yahoo-inc.com>
>
> On Jun 28, 2010, at 12:36 AM, Swati Jain wrote:
>
> Thanks for the prompt reply. As you mentioned optimization is in its
>> developing stage, does it mean optimization framework is not complete or
>> only rules are in developing stage? In addition to that, I would really
>> appreciate if you could give a rough idea when the patch will be available
>> and what functionality will it contain?
>>
>> At this point we believe the framework is complete and rules are being
> developed. But the framework has never been used in user testing situations
> (alpha or beta testing) so there will be a whole round of bugs to fix once
> that testing is done.
>
> The current plan is to switch to this code as the actual optimizer with
> 0.8, which we hope to release late this year (no promises).
>
> Alan.
>
Re: Bug in new logical optimizer framework?
Posted by Daniel Dai <ji...@yahoo-inc.com>.
The optimization code is in the package org.apache.pig.experimental. The
code is not used by default unless you set system property
pig.usenewlogicalplan to true. A good starting point might be
"org.apache.pig.experimental.logical.optimizer.LogicalPlanOptimizer".
Daniel
Renato Marroquín Mogrovejo wrote:
> Thanks Daniel and Dmitriy for your answers, now I have a much clear idea of
> what type of optimization work is being done on PIG. And just another quick
> question, do you guys know where in the code those optimizations are?
> I just want to give it a closer look (:
>
> Renato M.
>
> 2010/7/1 Dmitriy Ryaboy <dv...@gmail.com>
>
>
>> Renato,
>> I just want to make sure folks know -- Pig already has a number of such
>> optimizations. Daniel's work is aimed at making it (much) easier to write
>> such rules and to add a couple new ones. But some of the classic
>> optimizations like projection and filter push-down already exist in the
>> released versions of Pig.
>>
>> D
>>
>> On Thu, Jul 1, 2010 at 5:32 PM, Daniel Dai <ji...@yahoo-inc.com> wrote:
>>
>>
>>> Yes, they are classic logical optimization plus some Pig only
>>>
>> optimization.
>>
>>> All these are rule based.
>>>
>> https://issues.apache.org/jira/browse/PIG-1319is an umbrella Jira to track
>> all new optimization rules.
>>
>>> Daniel
>>>
>>>
>>> Renato Marroquín Mogrovejo wrote:
>>>
>>>
>>>> Hi,
>>>>
>>>> I am also interested in this logical plan optimization framework
>>>> functionality. You mentioned that rules are being developed, could you
>>>> explain a little bit more about them? Are they like the classic logical
>>>> optimizations (early projection, early filtering, among others)?
>>>> Thanks in advance.
>>>>
>>>> Renato M.
>>>>
>>>> logical plan optimization framework
>>>>
>>>> 2010/6/28 Alan Gates <ga...@yahoo-inc.com>
>>>>
>>>>
>>>>
>>>>
>>>>> On Jun 28, 2010, at 12:36 AM, Swati Jain wrote:
>>>>>
>>>>> Thanks for the prompt reply. As you mentioned optimization is in its
>>>>>
>>>>>
>>>>>
>>>>>> developing stage, does it mean optimization framework is not complete
>>>>>>
>> or
>>
>>>>>> only rules are in developing stage? In addition to that, I would
>>>>>>
>> really
>>
>>>>>> appreciate if you could give a rough idea when the patch will be
>>>>>> available
>>>>>> and what functionality will it contain?
>>>>>>
>>>>>> At this point we believe the framework is complete and rules are
>>>>>>
>> being
>>
>>>>>>
>>>>> developed. But the framework has never been used in user testing
>>>>> situations
>>>>> (alpha or beta testing) so there will be a whole round of bugs to fix
>>>>> once
>>>>> that testing is done.
>>>>>
>>>>> The current plan is to switch to this code as the actual optimizer with
>>>>> 0.8, which we hope to release late this year (no promises).
>>>>>
>>>>> Alan.
>>>>>
>>>>>
>>>>>
>>>>>
Re: Bug in new logical optimizer framework?
Posted by Renato Marroquín Mogrovejo <re...@gmail.com>.
Thanks Daniel and Dmitriy for your answers, now I have a much clear idea of
what type of optimization work is being done on PIG. And just another quick
question, do you guys know where in the code those optimizations are?
I just want to give it a closer look (:
Renato M.
2010/7/1 Dmitriy Ryaboy <dv...@gmail.com>
> Renato,
> I just want to make sure folks know -- Pig already has a number of such
> optimizations. Daniel's work is aimed at making it (much) easier to write
> such rules and to add a couple new ones. But some of the classic
> optimizations like projection and filter push-down already exist in the
> released versions of Pig.
>
> D
>
> On Thu, Jul 1, 2010 at 5:32 PM, Daniel Dai <ji...@yahoo-inc.com> wrote:
>
> > Yes, they are classic logical optimization plus some Pig only
> optimization.
> > All these are rule based.
> https://issues.apache.org/jira/browse/PIG-1319is an umbrella Jira to track
> all new optimization rules.
> >
> > Daniel
> >
> >
> > Renato Marroquín Mogrovejo wrote:
> >
> >> Hi,
> >>
> >> I am also interested in this logical plan optimization framework
> >> functionality. You mentioned that rules are being developed, could you
> >> explain a little bit more about them? Are they like the classic logical
> >> optimizations (early projection, early filtering, among others)?
> >> Thanks in advance.
> >>
> >> Renato M.
> >>
> >> logical plan optimization framework
> >>
> >> 2010/6/28 Alan Gates <ga...@yahoo-inc.com>
> >>
> >>
> >>
> >>> On Jun 28, 2010, at 12:36 AM, Swati Jain wrote:
> >>>
> >>> Thanks for the prompt reply. As you mentioned optimization is in its
> >>>
> >>>
> >>>> developing stage, does it mean optimization framework is not complete
> or
> >>>> only rules are in developing stage? In addition to that, I would
> really
> >>>> appreciate if you could give a rough idea when the patch will be
> >>>> available
> >>>> and what functionality will it contain?
> >>>>
> >>>> At this point we believe the framework is complete and rules are
> being
> >>>>
> >>>>
> >>> developed. But the framework has never been used in user testing
> >>> situations
> >>> (alpha or beta testing) so there will be a whole round of bugs to fix
> >>> once
> >>> that testing is done.
> >>>
> >>> The current plan is to switch to this code as the actual optimizer with
> >>> 0.8, which we hope to release late this year (no promises).
> >>>
> >>> Alan.
> >>>
> >>>
> >>>
> >>
> >
>
Re: Bug in new logical optimizer framework?
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Renato,
I just want to make sure folks know -- Pig already has a number of such
optimizations. Daniel's work is aimed at making it (much) easier to write
such rules and to add a couple new ones. But some of the classic
optimizations like projection and filter push-down already exist in the
released versions of Pig.
D
On Thu, Jul 1, 2010 at 5:32 PM, Daniel Dai <ji...@yahoo-inc.com> wrote:
> Yes, they are classic logical optimization plus some Pig only optimization.
> All these are rule based. https://issues.apache.org/jira/browse/PIG-1319is an umbrella Jira to track all new optimization rules.
>
> Daniel
>
>
> Renato Marroquín Mogrovejo wrote:
>
>> Hi,
>>
>> I am also interested in this logical plan optimization framework
>> functionality. You mentioned that rules are being developed, could you
>> explain a little bit more about them? Are they like the classic logical
>> optimizations (early projection, early filtering, among others)?
>> Thanks in advance.
>>
>> Renato M.
>>
>> logical plan optimization framework
>>
>> 2010/6/28 Alan Gates <ga...@yahoo-inc.com>
>>
>>
>>
>>> On Jun 28, 2010, at 12:36 AM, Swati Jain wrote:
>>>
>>> Thanks for the prompt reply. As you mentioned optimization is in its
>>>
>>>
>>>> developing stage, does it mean optimization framework is not complete or
>>>> only rules are in developing stage? In addition to that, I would really
>>>> appreciate if you could give a rough idea when the patch will be
>>>> available
>>>> and what functionality will it contain?
>>>>
>>>> At this point we believe the framework is complete and rules are being
>>>>
>>>>
>>> developed. But the framework has never been used in user testing
>>> situations
>>> (alpha or beta testing) so there will be a whole round of bugs to fix
>>> once
>>> that testing is done.
>>>
>>> The current plan is to switch to this code as the actual optimizer with
>>> 0.8, which we hope to release late this year (no promises).
>>>
>>> Alan.
>>>
>>>
>>>
>>
>
Re: Bug in new logical optimizer framework?
Posted by Daniel Dai <ji...@yahoo-inc.com>.
Yes, they are classic logical optimization plus some Pig only
optimization. All these are rule based.
https://issues.apache.org/jira/browse/PIG-1319 is an umbrella Jira to
track all new optimization rules.
Daniel
Renato Marroquín Mogrovejo wrote:
> Hi,
>
> I am also interested in this logical plan optimization framework
> functionality. You mentioned that rules are being developed, could you
> explain a little bit more about them? Are they like the classic logical
> optimizations (early projection, early filtering, among others)?
> Thanks in advance.
>
> Renato M.
>
> logical plan optimization framework
>
> 2010/6/28 Alan Gates <ga...@yahoo-inc.com>
>
>
>> On Jun 28, 2010, at 12:36 AM, Swati Jain wrote:
>>
>> Thanks for the prompt reply. As you mentioned optimization is in its
>>
>>> developing stage, does it mean optimization framework is not complete or
>>> only rules are in developing stage? In addition to that, I would really
>>> appreciate if you could give a rough idea when the patch will be available
>>> and what functionality will it contain?
>>>
>>> At this point we believe the framework is complete and rules are being
>>>
>> developed. But the framework has never been used in user testing situations
>> (alpha or beta testing) so there will be a whole round of bugs to fix once
>> that testing is done.
>>
>> The current plan is to switch to this code as the actual optimizer with
>> 0.8, which we hope to release late this year (no promises).
>>
>> Alan.
>>
>>