You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Pradeep Kamath (JIRA)" <ji...@apache.org> on 2009/09/05 00:52:58 UTC
[jira] Created: (PIG-946) Combiner optimizer does not optimize when
limit follow group, foreach
Combiner optimizer does not optimize when limit follow group, foreach
---------------------------------------------------------------------
Key: PIG-946
URL: https://issues.apache.org/jira/browse/PIG-946
Project: Pig
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Pradeep Kamath
The following script is combinable but is not optimized:
a = load '/user/pig/tests/data/singlefile/studenttab10k';
b = group a by $1;
c = foreach b generate group, AVG(a.$2);
d = limit c 10;
dump d;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-946) Combiner optimizer does not optimize when
limit follow group, foreach
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-946:
-------------------------------
Attachment: PIG-946-codechange-draft.patch
Attached patch has a first draft of the code changes required in CombinerOptiimizer.java. This is a non tested draft patch - needs to be augmented with testcases - attaching it merely for reference - should not be committed yet.
> Combiner optimizer does not optimize when limit follow group, foreach
> ---------------------------------------------------------------------
>
> Key: PIG-946
> URL: https://issues.apache.org/jira/browse/PIG-946
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.3.0
> Reporter: Pradeep Kamath
> Attachments: PIG-946-codechange-draft.patch
>
>
> The following script is combinable but is not optimized:
> a = load '/user/pig/tests/data/singlefile/studenttab10k';
> b = group a by $1;
> c = foreach b generate group, AVG(a.$2);
> d = limit c 10;
> dump d;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-946) Combiner optimizer does not optimize when
limit follow group, foreach
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-946:
---------------------------
Assignee: Thejas M Nair
Fix Version/s: 0.9.0
> Combiner optimizer does not optimize when limit follow group, foreach
> ---------------------------------------------------------------------
>
> Key: PIG-946
> URL: https://issues.apache.org/jira/browse/PIG-946
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.3.0
> Reporter: Pradeep Kamath
> Assignee: Thejas M Nair
> Fix For: 0.9.0
>
> Attachments: PIG-946-codechange-draft.patch
>
>
> The following script is combinable but is not optimized:
> a = load '/user/pig/tests/data/singlefile/studenttab10k';
> b = group a by $1;
> c = foreach b generate group, AVG(a.$2);
> d = limit c 10;
> dump d;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-946) Combiner optimizer does not optimize
when limit follow group, foreach
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751648#action_12751648 ]
Pradeep Kamath commented on PIG-946:
------------------------------------
The root cause is in the CombinerOptimizer, the code expects POPackage to have POForEach as the successor and then analyzes the POForEach to check if it can be combined. In the above query, the OpLimitOptimizer pushes the limit between the cogroup and foreach which in the MRPlan shows up as a POLimit between the POPackage and POForEach. In this case, the CombinerOptimizer should ignore the POLimit and still analyze the POForEach to check if combiner optimization is possible.
> Combiner optimizer does not optimize when limit follow group, foreach
> ---------------------------------------------------------------------
>
> Key: PIG-946
> URL: https://issues.apache.org/jira/browse/PIG-946
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.3.0
> Reporter: Pradeep Kamath
>
> The following script is combinable but is not optimized:
> a = load '/user/pig/tests/data/singlefile/studenttab10k';
> b = group a by $1;
> c = foreach b generate group, AVG(a.$2);
> d = limit c 10;
> dump d;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.