You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "rahul k singh (JIRA)" <ji...@apache.org> on 2009/10/23 12:37:59 UTC
[jira] Created: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
runningMapTasks counter is not properly decremented in case of failed Tasks.
----------------------------------------------------------------------------
Key: MAPREDUCE-1143
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: rahul k singh
Priority: Blocker
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-7.patch
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Patch Available (was: Open)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789175#action_12789175 ]
Hemanth Yamijala commented on MAPREDUCE-1143:
---------------------------------------------
Arun helped us understand the change that was discussed and I really like the idea. Rather than looking at this as a way of the TIP calling into the JIP to update counters, I started thinking about this as the TIP deciding what has changed in its state and informing about the change to the JIP. Then JIP decides to update its state (basically the counters) based on this information. This is a really good separation of ownership.
However, Arun, Rahul and I looked into making this change into the current code base and fell out of confidence with respect to the level of change it was causing. Given this point, we now feel that my original suggestion [here|https://issues.apache.org/jira/browse/MAPREDUCE-1143?focusedCommentId=12784077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12784077], while agreeably not as good as Owen's, is a *safer* and more localized change to implement than the alternate proposal. And we suggest taking that approach to fix this JIRA.
A rewrite of this status update code is really long overdue, and it would anyway be done in the manner Owen is proposing, but in a separate JIRA. Hope this is agreeable to all.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789220#action_12789220 ]
Arun C Murthy commented on MAPREDUCE-1143:
------------------------------------------
Clearly +1 to Hemanth's suggestion from my end since I'm part of the cabal... *smile*
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Open (was: Patch Available)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791775#action_12791775 ]
rahul k singh commented on MAPREDUCE-1143:
------------------------------------------
Ran TestStreamingExitStatus runs independently fine on my local box.
Looked at the error, the failure is not related to this patch.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Open (was: Patch Available)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-ydist-8.patch.txt
metrics related counters were removed from trunk as part of 1152 , hence they are not in the fix for trunk , For yahoo distribution metric is still in there in failedTask.
implemented suggestions by arun.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-ydist-4.patch
Changed the patch to handle changes to work with ydist branch
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-4.patch
added metering directly under wasRunning
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790108#action_12790108 ]
rahul k singh commented on MAPREDUCE-1143:
------------------------------------------
most of the test cases are failing in the build.
with
org.apache.hadoop.ipc.RPC.waitForProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/ipc/VersionedProtocol;
java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.waitForProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/ipc/VersionedProtocol;
This is happening on the trunk too , i.e , without applying the patch . This is in no way related to the current fix.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Patch Available (was: Open)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned MAPREDUCE-1143:
----------------------------------------
Assignee: rahul k singh
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Patch Available (was: Open)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-ydist-7.patch
Incorporate the hemanth's suggestions
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790003#action_12790003 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1143:
----------------------------------------------------
I have the same comment as at [this|https://issues.apache.org/jira/browse/MAPREDUCE-1143?focusedCommentId=12770877&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12770877]. Shouldn't meterTaskAttempt() be in wasAttemptRunning check ?
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769174#action_12769174 ]
rahul k singh commented on MAPREDUCE-1143:
------------------------------------------
We increment runningMapTasks counter every time , we give a task speculative or non-speculative . This counter is not decremented properly in case there is a failed task while of the speculative attempts is alive. This is because
{code}
if(!tip.isRunning()) {
decrement running counter;
}
{code}
{{tip.isRunning()}} would keep on returning true if any of the attempts in the {{tip}} are still running.
This would mean that for each task , we would only decrement once , whereas we are incrementing the counter for every attempt of the task.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-v21.patch
patch for 21
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-v21.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Patch Available (was: Open)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Patch Available (was: Open)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-ydist-9.patch
incorporated amareshwari's comment
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-5.patch.txt
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-2.patch
Attaching the new patch ,
There was small correction in terms of usage of wasRunning.
In the new patch we decrement counters when wasRunning is true.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Release Note: Corrects the behaviour of tasks counters in case of failed tasks.Incorrect counter values can lead to bad scheduling decisions .This jira rectifies the problem by making sure decrement properly happens incase of failed tasks. (was: Corrects the behaviour of tasks counters in case of failed tasks,incorrect counter values can lead to bad scheduling decisions .This jira provides the fix of incorrect behaviour and in result enhancing the scheduling. )
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-v21.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790010#action_12790010 ]
rahul k singh commented on MAPREDUCE-1143:
------------------------------------------
As hemanth mentioned above [here|https://issues.apache.org/jira/browse/MAPREDUCE-1143?focusedCommentId=12789175&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12789175] , for this jira , the changes are confined to runinng count in the trunk.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796726#action_12796726 ]
Hudson commented on MAPREDUCE-1143:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/])
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-v21.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Open (was: Patch Available)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-6.patch
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Open (was: Patch Available)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790022#action_12790022 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1143:
----------------------------------------------------
meterTaskAttempt() is gathering statistics for an attempt that just finished, I think it should also be in wasAttemptRunning check.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Patch Available (was: Open)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771071#action_12771071 ]
Hadoop QA commented on MAPREDUCE-1143:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12423438/MAPRED-1143-2.patch
against trunk revision 830531.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/99/console
This message is automatically generated.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790708#action_12790708 ]
rahul k singh commented on MAPREDUCE-1143:
------------------------------------------
ran ant test-contrib.
TestStreamingExitStatus was failing , but it is failing without the patch too.
rest all tests passed .
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Patch Available (was: Open)
checking with hudson
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785199#action_12785199 ]
Devaraj Das commented on MAPREDUCE-1143:
----------------------------------------
Yes the suggestion is to increment the counters in the TIP. This is not the ideal solution but it seems like doing the counter updates there via accessing the same from the corresponding JIP would make the updates very precise...
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778233#action_12778233 ]
Hemanth Yamijala commented on MAPREDUCE-1143:
---------------------------------------------
I spoke to Amarsri and Rahul about my comments and found out some explanations:
bq. For instance, even after this patch, I see that the number of running tasks is decremented under different checks when a task completes and when a task fails. I assume this is for good reason, but still it is difficult to review.
So, the different checks are as follows:
{code}
completedTask() {
if (this tip is complete) {
return;
}
update counters
}
failedTask() {
if (any attempt was running for this tip before status update) {
update counters
}
}
{code}
It appears completedTask doesn't need the check for TIP being complete at all, as it can never happen. A tip is marked complete only if atleast one attempt has completed and remains so. If another attempt comes in reporting success now, we fail this in status update and do not follow the completedTask code path at all. So, for all practical purposes, counters are being updated unconditionally in completedTask. Further, in the same code path, the task is removed from the active tasks as well. Hence no further check is necessary.
The check in failedTask is required though. This is because a task can fail *after* it has been marked as succeeded. For e.g. if there are fetch failures for a map, or if a tracker is lost. In this case, we should not update counters again because they would have already been updated when the task succeeded.
However, in this context, I am a little worried that we are checking for any attempt being running before status update, rather than this specific attempt. At least in theory it is possible this results in some inconsistency.
Consider this sequence of events:
- A task is scheduled
- It is speculated
- It completes -> Counters are decremented here.
- It fails (lost TT, fetch failures) -> The current patch will decrement counters here again.
- The speculated attempt succeeds.
In practice though, this scenario may not be very likely. Apparently fetch failures and lost TTs are the only extreme cases when this is possible. And there is considerable time lag that can happen before a task completes and it has to be failed. The time lag will in most cases be large enough to kill the speculative attempt as well.
With this background, is it worth changing the current patch to:
{code}
failedTask() {
if (this task was running before status update) {
update counters
}
}
{code}
This seems more correct to me, but was wondering if it was worth the change. Thoughts ?
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790065#action_12790065 ]
Hadoop QA commented on MAPREDUCE-1143:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427890/MAPRED-1143-6.patch
against trunk revision 889786.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/190/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/190/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/190/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/190/console
This message is automatically generated.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-1143:
-------------------------------------
Status: Open (was: Patch Available)
Had an offline discussion with Owen and Devaraj about this mess...
Owen suggested a better, albeit more involved, fix:
Update the counters in TIP.updateStatus since it's the single entry point for updating TaskStatus for a task-attempt and since JIP is already locked we reach into into and update runningMapTasks, runningReduceTasks, failedMapTasks, failedReduceTasks, speculativeMapTasks and speculativeReduceTasks.
Thoughts?
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789324#action_12789324 ]
Hadoop QA commented on MAPREDUCE-1143:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427709/MAPRED-1143-5.patch.txt
against trunk revision 889571.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/320/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/320/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/320/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/320/console
This message is automatically generated.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-ydist-2.patch
yahoo dist patch --no-prefix
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791752#action_12791752 ]
Hadoop QA commented on MAPREDUCE-1143:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427899/MAPRED-1143-7.patch
against trunk revision 891111.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/205/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/205/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/205/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/205/console
This message is automatically generated.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Patch Available (was: Open)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790709#action_12790709 ]
rahul k singh commented on MAPREDUCE-1143:
------------------------------------------
the above comment is w.r.t to yahoo hadoop 20 version
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792118#action_12792118 ]
Hadoop QA commented on MAPREDUCE-1143:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427899/MAPRED-1143-7.patch
against trunk revision 891524.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/211/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/211/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/211/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/211/console
This message is automatically generated.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772452#action_12772452 ]
Hadoop QA commented on MAPREDUCE-1143:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12423665/MAPRED-1143-4.patch
against trunk revision 831037.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/117/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/117/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/117/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/117/console
This message is automatically generated.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-ydist-3.patch
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784077#action_12784077 ]
Hemanth Yamijala commented on MAPREDUCE-1143:
---------------------------------------------
After a discussion with Arun, I felt I might clarify little more on what I am proposing. Some details:
- In TaskInProgress.java, introduce:
{code}
boolean isRunning(TaskAttemptID taskId) {
return activeTasks.containsKey(taskId);
}
{code}
- Modify JobInProgress.failedTask private API to have an additional parameter wasAttemptRunning, which would be initialized in JIP.updateTaskStatus to tip.isRunning(status.getTaskID())
- Use wasAttemptRunning only to update the running* counters
I originally thought we can modify wasRunning to indicate if the attempt was running (rather than if the TIP was running). But after speaking with Arun, I feel we want to localize the changes to as much as possible.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771848#action_12771848 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1143:
----------------------------------------------------
changes look fine to me.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-1.patch
Attaching the patch with the test case.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785177#action_12785177 ]
Hemanth Yamijala commented on MAPREDUCE-1143:
---------------------------------------------
If I understand correctly, you are suggesting we expose the counters in JIP for updates from TIP, right ? I feel conceptually, TIP does not own the counters, JIP does. Hence, I feel updates to these must be done by JIP and not TIP. Wouldn't you agree with this ?
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790078#action_12790078 ]
Arun C Murthy commented on MAPREDUCE-1143:
------------------------------------------
+1 for the patch.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Patch Available (was: Open)
checking with hudson
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790638#action_12790638 ]
rahul k singh commented on MAPREDUCE-1143:
------------------------------------------
ran ant test for yhadoop version , the test TestJobHistory failed , i re ran it locally it passed.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770877#action_12770877 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1143:
----------------------------------------------------
Sorry, second if check looks fine.
But meterTaskAttempt is calculating statistics. I think It should be outside the !isRunning check.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Release Note: Corrects the behaviour of tasks counters in case of failed tasks,incorrect counter values can lead to bad scheduling decisions .This jira provides the fix of incorrect behaviour and in result enhancing the scheduling.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-v21.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790165#action_12790165 ]
rahul k singh commented on MAPREDUCE-1143:
------------------------------------------
tested the fix for yhadoop , manually, it works fine.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774942#action_12774942 ]
Hemanth Yamijala commented on MAPREDUCE-1143:
---------------------------------------------
Not really about this patch, but the logic for task status updates is way too complicated to reason correctly. For instance, even after this patch, I see that the number of running tasks is decremented under different checks when a task completes and when a task fails. I assume this is for good reason, but still it is difficult to review.
A radically different idea, which might be completely wrong (*smile*):
Can we assume this invariant:
- A task attempt is added to the list of active tasks in TaskInProgress whenever it is scheduled. Running task counts should be incremented in this code path.
- Likewise, a decrement of running tasks should happen whenever the task attempt being reported as completed or failed is removed from the active tasks list.
At least to me, this appears simpler to understand, and fixing the issue reported. Does this make sense ? Anything wrong in the logic ?
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-1143:
-------------------------------------
Status: Open (was: Patch Available)
Rahul, Amareshwari is right... it's reasonable to move the call to meterTaskAttempt too.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-1143:
----------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this to trunk and branch 0.21. Thanks, Rahul !
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-v21.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-1143:
-------------------------------------
Affects Version/s: 0.20.1
Fix Version/s: 0.21.0
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792384#action_12792384 ]
Hemanth Yamijala commented on MAPREDUCE-1143:
---------------------------------------------
+1 for the 21 patch. I will commit this.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-v21.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771431#action_12771431 ]
Hadoop QA commented on MAPREDUCE-1143:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12423564/MAPRED-1143-3.patch
against trunk revision 830837.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/103/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/103/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/103/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/103/console
This message is automatically generated.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770895#action_12770895 ]
rahul k singh commented on MAPREDUCE-1143:
------------------------------------------
after discussing the meterTaskAttempt issue internally it was decided to leave it as is .
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-2.patch
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-ydist-1.patch
yahoo dist
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-ydist-1.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-ydist-6.patch
a small nit in the earlier ydist patch , rectified in this
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771320#action_12771320 ]
Jothi Padmanabhan commented on MAPREDUCE-1143:
----------------------------------------------
Vert minor nits with the test case, since you need to regenerate the patch anyway:
# I do not think {{job.finishTask(taskAttemptID[6])}} is required as that attempt is already failed
# Can we rename oldRunningReduce to oldRunningReduces
# Do we need to to {{clock.advance(20000)}} before {{clock.advanceBySpeculativeLag()}}, just checking..
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-3.patch
Changed to patch to apply on trunk and added comments by jothi
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Attachment: MAPRED-1143-ydist-5.patch
Attached the new patch moved metering along with other counters.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated MAPREDUCE-1143:
-------------------------------------
Status: Open (was: Patch Available)
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790122#action_12790122 ]
Hadoop QA commented on MAPREDUCE-1143:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427899/MAPRED-1143-7.patch
against trunk revision 889786.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/325/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/325/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/325/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/325/console
This message is automatically generated.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790581#action_12790581 ]
rahul k singh commented on MAPREDUCE-1143:
------------------------------------------
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
[exec]
[exec]
[exec]
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================
[exec]
[exec]
This fix doesnt have testcase , as it requires a significant change , we have provided for the trunk , manual testing is done to make sure that things are fine. for eclipse classpath its a known issue for 20 internal patch.
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-1143:
-------------------------------------
Status: Open (was: Patch Available)
The trunk and ydist patch differ in one significant manner:
{noformat}
+ if(wasAttemptRunning) {
+ if(!tip.isJobCleanupTask() && !tip.isJobSetupTask()) {
+ if(tip.isMapTask()) {
+ runningMapTasks -= 1;
+ } else {
+ runningReduceTasks -= 1;
+ }
+ }
+ }
{noformat}
{noformat}
boolean isComplete = tip.isComplete();
+ if (wasAttemptRunning) {
+ if (!tip.isJobCleanupTask() && !tip.isJobSetupTask()) {
+ if (tip.isMapTask()) {
+ runningMapTasks -= 1;
+ metrics.failedMap(taskid);
+ } else {
+ runningReduceTasks -= 1;
+ metrics.failedReduce(taskid);
+ }
+ }
+ }
{noformat}
----
Minor nits:
# Spacing: call to JIP.failedTask needs a space for the new parameter
# The above block of code could use a comment
# I'd appreciate if the following change was minimized by adding the new parameter to a new line by itself without changing the others:
{noformat}
- private void failedTask(TaskInProgress tip, TaskAttemptID taskid,
- TaskStatus status,
- TaskTracker taskTracker,
- boolean wasRunning, boolean wasComplete) {
+ private void failedTask(
+ TaskInProgress tip, TaskAttemptID taskid, TaskStatus status,
+ TaskTracker taskTracker, boolean wasRunning, boolean wasComplete,
+ boolean wasAttemptRunning) {
{noformat}
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790111#action_12790111 ]
rahul k singh commented on MAPREDUCE-1143:
------------------------------------------
opened jira [1292|https://issues.apache.org/jira/browse/MAPREDUCE-1292] for the above issue
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: rahul k singh
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not
properly decremented in case of failed Tasks.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770850#action_12770850 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1143:
----------------------------------------------------
second if block should still check for wasRunning && !isRunning
> runningMapTasks counter is not properly decremented in case of failed Tasks.
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1143
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: rahul k singh
> Priority: Blocker
> Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.