You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Thejas M Nair (JIRA)" <ji...@apache.org> on 2010/09/17 02:34:33 UTC
[jira] Created: (PIG-1617) 'group all' should always use one
reducer
'group all' should always use one reducer
-----------------------------------------
Key: PIG-1617
URL: https://issues.apache.org/jira/browse/PIG-1617
Project: Pig
Issue Type: Improvement
Affects Versions: 0.8.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Fix For: 0.8.0
'group all' sends all rows to a single reducer, it does not make sense to spawn more than one reducer for it. But if higher value of parallelism is specified or if the input is large enough so that changes in PIG-1249 result in larger value being set, there are additional reducers spawned that don't do anything useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1617) 'group all' should always use one
reducer
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-1617:
-------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
Patch committed to trunk and 0.8 branch.
> 'group all' should always use one reducer
> -----------------------------------------
>
> Key: PIG-1617
> URL: https://issues.apache.org/jira/browse/PIG-1617
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1617.1.patch
>
>
> 'group all' sends all rows to a single reducer, it does not make sense to spawn more than one reducer for it. But if higher value of parallelism is specified or if the input is large enough so that changes in PIG-1249 result in larger value being set, there are additional reducers spawned that don't do anything useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1617) 'group all' should always use one
reducer
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-1617:
-------------------------------
Status: Patch Available (was: Open)
> 'group all' should always use one reducer
> -----------------------------------------
>
> Key: PIG-1617
> URL: https://issues.apache.org/jira/browse/PIG-1617
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1617.1.patch
>
>
> 'group all' sends all rows to a single reducer, it does not make sense to spawn more than one reducer for it. But if higher value of parallelism is specified or if the input is large enough so that changes in PIG-1249 result in larger value being set, there are additional reducers spawned that don't do anything useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1617) 'group all' should always use one
reducer
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12912655#action_12912655 ]
Olga Natkovich commented on PIG-1617:
-------------------------------------
Looks good. +1
> 'group all' should always use one reducer
> -----------------------------------------
>
> Key: PIG-1617
> URL: https://issues.apache.org/jira/browse/PIG-1617
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1617.1.patch
>
>
> 'group all' sends all rows to a single reducer, it does not make sense to spawn more than one reducer for it. But if higher value of parallelism is specified or if the input is large enough so that changes in PIG-1249 result in larger value being set, there are additional reducers spawned that don't do anything useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1617) 'group all' should always use one
reducer
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-1617:
-------------------------------
Attachment: PIG-1617.1.patch
PIG-1617.1.patch- Patch sets parallelism of LOCogroup to 1 for group on a constant (including 'group all')
Passes unit tests and test patch. Ready for review.
> 'group all' should always use one reducer
> -----------------------------------------
>
> Key: PIG-1617
> URL: https://issues.apache.org/jira/browse/PIG-1617
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1617.1.patch
>
>
> 'group all' sends all rows to a single reducer, it does not make sense to spawn more than one reducer for it. But if higher value of parallelism is specified or if the input is large enough so that changes in PIG-1249 result in larger value being set, there are additional reducers spawned that don't do anything useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.