You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2009/01/12 19:04:00 UTC
[jira] Created: (PIG-615) Wrong number of jobs with limit
Wrong number of jobs with limit
-------------------------------
Key: PIG-615
URL: https://issues.apache.org/jira/browse/PIG-615
Project: Pig
Issue Type: Bug
Reporter: Olga Natkovich
Assignee: Shravan Matthur Narayanamurthy
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-615) Wrong number of jobs with limit
Posted by "Shravan Matthur Narayanamurthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shravan Matthur Narayanamurthy updated PIG-615:
-----------------------------------------------
Status: Patch Available (was: Open)
Here is the change I suggested as a patch. Hope this is what was expected
> Wrong number of jobs with limit
> -------------------------------
>
> Key: PIG-615
> URL: https://issues.apache.org/jira/browse/PIG-615
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shravan Matthur Narayanamurthy
> Attachments: 615.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-615) Wrong number of jobs with limit
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Santhosh Srinivasan updated PIG-615:
------------------------------------
Attachment: 615_1.patch
Attached patch includes Shravan's fix along with test cases that I added. Running unit test cases now.
> Wrong number of jobs with limit
> -------------------------------
>
> Key: PIG-615
> URL: https://issues.apache.org/jira/browse/PIG-615
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shravan Matthur Narayanamurthy
> Attachments: 615.patch, 615_1.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-615) Wrong number of jobs with limit
Posted by "Shravan Matthur Narayanamurthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665077#action_12665077 ]
Shravan Matthur Narayanamurthy commented on PIG-615:
----------------------------------------------------
Should I submit the changes I suggested in our discussion as a patch?
A summary of the discussion follows:
{quote}
As per the current logic, the generation of the 4th MR Job in case of limit depends on the use of *"parallel"* keyword.
Though the logic is not directly dependent on cluster configuration, some cluster configs require this 4th MR job and some don't.
For ex., if the cluster is configured to set number of reducers to one if parallelism is -1 or unspecified then our current logic will work as the 4th MR Job is redundant.
However, if the cluster is configured to set number of reducers to some other number, like 0.9 times the number of reduce slots if parallelism is unspecified then the 4th MRJob is necessary.
That is, we are making an implicit assumption in the code that if parallel is not explicitly mentioned, then the number of reducers is equal to 1. So the logic needs to be changed to include the 4th MRJob whenever the parallelism is not explicitly set to 1 so that we will produce correct results though in some cases its use might be redundant.
{quote}
> Wrong number of jobs with limit
> -------------------------------
>
> Key: PIG-615
> URL: https://issues.apache.org/jira/browse/PIG-615
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shravan Matthur Narayanamurthy
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-615) Wrong number of jobs with limit
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Santhosh Srinivasan updated PIG-615:
------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
All unit test cases passed. Patch has been committed. Thanks for the fix Shravan.
> Wrong number of jobs with limit
> -------------------------------
>
> Key: PIG-615
> URL: https://issues.apache.org/jira/browse/PIG-615
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shravan Matthur Narayanamurthy
> Attachments: 615.patch, 615_1.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-615) Wrong number of jobs with limit
Posted by "Shravan Matthur Narayanamurthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shravan Matthur Narayanamurthy updated PIG-615:
-----------------------------------------------
Attachment: 615.patch
> Wrong number of jobs with limit
> -------------------------------
>
> Key: PIG-615
> URL: https://issues.apache.org/jira/browse/PIG-615
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shravan Matthur Narayanamurthy
> Attachments: 615.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-615) Wrong number of jobs with limit
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12666654#action_12666654 ]
Santhosh Srinivasan commented on PIG-615:
-----------------------------------------
I will be reviewing this patch.
> Wrong number of jobs with limit
> -------------------------------
>
> Key: PIG-615
> URL: https://issues.apache.org/jira/browse/PIG-615
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shravan Matthur Narayanamurthy
> Attachments: 615.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.