You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Ashutosh Chauhan (JIRA)" <ji...@apache.org> on 2009/09/10 05:43:58 UTC
[jira] Created: (PIG-951) Reset parallelism to 1 for indexing job
in MergeJoin
Reset parallelism to 1 for indexing job in MergeJoin
----------------------------------------------------
Key: PIG-951
URL: https://issues.apache.org/jira/browse/PIG-951
Project: Pig
Issue Type: Bug
Components: impl
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
After sampling one tuple from every block, one reducer is used to sort the index entries in reduce phase to produce sorted index to be used in actual join job. Thus, parallelism of index job should be explictly set to 1. Currently, its not.
Currently, this is a non-issue, since we don't allow any blocking operators in pipeline before merge-join. However, later when we do allow blocking operators, then parallelism of indexing job will be that of preceding blocking operator. Even then, job will complete successfully because all tuple will go to only one reducer, because we are grouping on only one key "all". However, it will waste cluster resources by starting all the extra reducers which get no data and thus do nothing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-951) Reset parallelism to 1 for indexing job
in MergeJoin
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753941#action_12753941 ]
Hadoop QA commented on PIG-951:
-------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419132/pig-951.patch
against trunk revision 813601.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/23/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/23/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/23/console
This message is automatically generated.
> Reset parallelism to 1 for indexing job in MergeJoin
> ----------------------------------------------------
>
> Key: PIG-951
> URL: https://issues.apache.org/jira/browse/PIG-951
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Attachments: pig-951.patch
>
>
> After sampling one tuple from every block, one reducer is used to sort the index entries in reduce phase to produce sorted index to be used in actual join job. Thus, parallelism of index job should be explictly set to 1. Currently, its not.
> Currently, this is a non-issue, since we don't allow any blocking operators in pipeline before merge-join. However, later when we do allow blocking operators, then parallelism of indexing job will be that of preceding blocking operator. Even then, job will complete successfully because all tuple will go to only one reducer, because we are grouping on only one key "all". However, it will waste cluster resources by starting all the extra reducers which get no data and thus do nothing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-951) Reset parallelism to 1 for indexing job
in MergeJoin
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-951:
---------------------------------
Attachment: pig-951.patch
One line patch which fixes this. Also, added test case to catch regression on this.
> Reset parallelism to 1 for indexing job in MergeJoin
> ----------------------------------------------------
>
> Key: PIG-951
> URL: https://issues.apache.org/jira/browse/PIG-951
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Attachments: pig-951.patch
>
>
> After sampling one tuple from every block, one reducer is used to sort the index entries in reduce phase to produce sorted index to be used in actual join job. Thus, parallelism of index job should be explictly set to 1. Currently, its not.
> Currently, this is a non-issue, since we don't allow any blocking operators in pipeline before merge-join. However, later when we do allow blocking operators, then parallelism of indexing job will be that of preceding blocking operator. Even then, job will complete successfully because all tuple will go to only one reducer, because we are grouping on only one key "all". However, it will waste cluster resources by starting all the extra reducers which get no data and thus do nothing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-951) Reset parallelism to 1 for indexing job
in MergeJoin
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756654#action_12756654 ]
Alan Gates commented on PIG-951:
--------------------------------
I'll be reviewing this patch.
> Reset parallelism to 1 for indexing job in MergeJoin
> ----------------------------------------------------
>
> Key: PIG-951
> URL: https://issues.apache.org/jira/browse/PIG-951
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Attachments: pig-951.patch
>
>
> After sampling one tuple from every block, one reducer is used to sort the index entries in reduce phase to produce sorted index to be used in actual join job. Thus, parallelism of index job should be explictly set to 1. Currently, its not.
> Currently, this is a non-issue, since we don't allow any blocking operators in pipeline before merge-join. However, later when we do allow blocking operators, then parallelism of indexing job will be that of preceding blocking operator. Even then, job will complete successfully because all tuple will go to only one reducer, because we are grouping on only one key "all". However, it will waste cluster resources by starting all the extra reducers which get no data and thus do nothing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-951) Reset parallelism to 1 for indexing job
in MergeJoin
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-951:
-------------------------------
Status: Patch Available (was: Open)
> Reset parallelism to 1 for indexing job in MergeJoin
> ----------------------------------------------------
>
> Key: PIG-951
> URL: https://issues.apache.org/jira/browse/PIG-951
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Attachments: pig-951.patch
>
>
> After sampling one tuple from every block, one reducer is used to sort the index entries in reduce phase to produce sorted index to be used in actual join job. Thus, parallelism of index job should be explictly set to 1. Currently, its not.
> Currently, this is a non-issue, since we don't allow any blocking operators in pipeline before merge-join. However, later when we do allow blocking operators, then parallelism of indexing job will be that of preceding blocking operator. Even then, job will complete successfully because all tuple will go to only one reducer, because we are grouping on only one key "all". However, it will waste cluster resources by starting all the extra reducers which get no data and thus do nothing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-951) Reset parallelism to 1 for indexing job
in MergeJoin
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-951:
---------------------------
Resolution: Fixed
Fix Version/s: 0.6.0
Status: Resolved (was: Patch Available)
Patch checked in. Thanks Ashutosh.
> Reset parallelism to 1 for indexing job in MergeJoin
> ----------------------------------------------------
>
> Key: PIG-951
> URL: https://issues.apache.org/jira/browse/PIG-951
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.6.0
>
> Attachments: pig-951.patch
>
>
> After sampling one tuple from every block, one reducer is used to sort the index entries in reduce phase to produce sorted index to be used in actual join job. Thus, parallelism of index job should be explictly set to 1. Currently, its not.
> Currently, this is a non-issue, since we don't allow any blocking operators in pipeline before merge-join. However, later when we do allow blocking operators, then parallelism of indexing job will be that of preceding blocking operator. Even then, job will complete successfully because all tuple will go to only one reducer, because we are grouping on only one key "all". However, it will waste cluster resources by starting all the extra reducers which get no data and thus do nothing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.