You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Pradeep Kamath (JIRA)" <ji...@apache.org> on 2009/01/31 00:20:59 UTC
[jira] Created: (PIG-651) PERFORMANCE: Use specialized
POForEachNoFlatten for cases where the foreach has no flattens
PERFORMANCE: Use specialized POForEachNoFlatten for cases where the foreach has no flattens
-------------------------------------------------------------------------------------------
Key: PIG-651
URL: https://issues.apache.org/jira/browse/PIG-651
Project: Pig
Issue Type: Improvement
Affects Versions: types_branch
Reporter: Pradeep Kamath
Assignee: Pradeep Kamath
Fix For: types_branch
POForEach has lot of code to handle flattening (cross product) of the fields in the generate. This is relevant only when atleast one field in the generate needs to be flattened. If all fields in the generate do not need to be flattened, a more simplified and hopefully more efficient POForEach can be used.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-651) PERFORMANCE: Use specialized
POForEachNoFlatten for cases where the foreach has no flattens
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12757297#action_12757297 ]
Alan Gates commented on PIG-651:
--------------------------------
Is it worth adding this complexity to the code for a 2% speed up? I'd vote no.
> PERFORMANCE: Use specialized POForEachNoFlatten for cases where the foreach has no flattens
> -------------------------------------------------------------------------------------------
>
> Key: PIG-651
> URL: https://issues.apache.org/jira/browse/PIG-651
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.2.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: PIG-651.patch
>
>
> POForEach has lot of code to handle flattening (cross product) of the fields in the generate. This is relevant only when atleast one field in the generate needs to be flattened. If all fields in the generate do not need to be flattened, a more simplified and hopefully more efficient POForEach can be used.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-651) PERFORMANCE: Use specialized
POForEachNoFlatten for cases where the foreach has no flattens
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-651:
---------------------------
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
> PERFORMANCE: Use specialized POForEachNoFlatten for cases where the foreach has no flattens
> -------------------------------------------------------------------------------------------
>
> Key: PIG-651
> URL: https://issues.apache.org/jira/browse/PIG-651
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.2.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: PIG-651.patch
>
>
> POForEach has lot of code to handle flattening (cross product) of the fields in the generate. This is relevant only when atleast one field in the generate needs to be flattened. If all fields in the generate do not need to be flattened, a more simplified and hopefully more efficient POForEach can be used.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-651) PERFORMANCE: Use specialized
POForEachNoFlatten for cases where the foreach has no flattens
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-651:
-------------------------------
Attachment: PIG-651.patch
> PERFORMANCE: Use specialized POForEachNoFlatten for cases where the foreach has no flattens
> -------------------------------------------------------------------------------------------
>
> Key: PIG-651
> URL: https://issues.apache.org/jira/browse/PIG-651
> Project: Pig
> Issue Type: Improvement
> Affects Versions: types_branch
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: types_branch
>
> Attachments: PIG-651.patch
>
>
> POForEach has lot of code to handle flattening (cross product) of the fields in the generate. This is relevant only when atleast one field in the generate needs to be flattened. If all fields in the generate do not need to be flattened, a more simplified and hopefully more efficient POForEach can be used.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-651) PERFORMANCE: Use specialized
POForEachNoFlatten for cases where the foreach has no flattens
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12757305#action_12757305 ]
Olga Natkovich commented on PIG-651:
------------------------------------
I agree that there is not enough gain
> PERFORMANCE: Use specialized POForEachNoFlatten for cases where the foreach has no flattens
> -------------------------------------------------------------------------------------------
>
> Key: PIG-651
> URL: https://issues.apache.org/jira/browse/PIG-651
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.2.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: PIG-651.patch
>
>
> POForEach has lot of code to handle flattening (cross product) of the fields in the generate. This is relevant only when atleast one field in the generate needs to be flattened. If all fields in the generate do not need to be flattened, a more simplified and hopefully more efficient POForEach can be used.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-651) PERFORMANCE: Use specialized
POForEachNoFlatten for cases where the foreach has no flattens
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-651:
-------------------------------
Status: Patch Available (was: Open)
Attached patch implements a simpler POForEachNoFlatten whenever no flattens are present in the POForEach (this is determined in LogToPhyTranslator and is also used in CombinerOptimizer for the map and combine plan foreachs). Initial tests show marginal speedup. (10 seconds out of 9 mins 50 secs in one particular group by query).
> PERFORMANCE: Use specialized POForEachNoFlatten for cases where the foreach has no flattens
> -------------------------------------------------------------------------------------------
>
> Key: PIG-651
> URL: https://issues.apache.org/jira/browse/PIG-651
> Project: Pig
> Issue Type: Improvement
> Affects Versions: types_branch
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: types_branch
>
> Attachments: PIG-651.patch
>
>
> POForEach has lot of code to handle flattening (cross product) of the fields in the generate. This is relevant only when atleast one field in the generate needs to be flattened. If all fields in the generate do not need to be flattened, a more simplified and hopefully more efficient POForEach can be used.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.