[jira] Resolved: (PIG-273) Need to optimize the ways splits are handled, both in the top level plan and in nested plans.

["Alan Gates (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alan Gates resolved PIG-273.
----------------------------

    Resolution: Fixed

Inner plans are not optimal in that they are cloned.  But I don't think we're going to solve this anytime soon, and we are not seeing users complain about lack of performance in this area.  Optimizing of external splits is being done in PIG-627.  So I'm closing this bug.

> Need to optimize the ways splits are handled, both in the top level plan and in nested plans.
> ---------------------------------------------------------------------------------------------
>
>                 Key: PIG-273
>                 URL: https://issues.apache.org/jira/browse/PIG-273
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Alan Gates
>            Priority: Minor
>
> Currently, in the new pipeline rework (see PIG-157), splits in the data flow are not handled efficiently.  
> In the top level plans splits cause all the output data to be written to hdfs and then reread by each leg of the split.  This forces both a read/write and a new map/reduce pass when it is not always necessary.  For example, consider:
> A = load 'myfile';
> split A into B  if $0 < 100, C if $0 >= 100;
> B1 = group B by $0;
> ...
> C1 = group B by $1;
> ...
> In this case A will be loaded, and then immediately stored again.  Then a plan will be executed that handles the B* part of the script, and then another executed that will handle the C* part of the script.
> In nested plans, each projection of the generate is computed separately, even if they share common steps in the plan.  For example:
> B = group A by $0;
> C= foreach B {
>     C1 = distinct $1;
>     C2 = filter C1 by $1 > 0;
>     generate group, COUNT(C1), COUNT(C2);
> }
> That will currently be executed with two nested plans, distinct->COUNT(C1) and distinct->filter->COUNT(C2).  The same distinct will be computed twice.  Ideally we would like to compute the distinct once and then split the output.
> I suspect that optimizing the inner plan is more important because there are more situations where this occurs.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.