You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2009/04/30 20:59:30 UTC
[jira] Created: (HADOOP-5761) JobTracker and TaskTracker enter
infinite loop when TaskTracker reports bad taskid
JobTracker and TaskTracker enter infinite loop when TaskTracker reports bad taskid
----------------------------------------------------------------------------------
Key: HADOOP-5761
URL: https://issues.apache.org/jira/browse/HADOOP-5761
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Todd Lipcon
If the TaskTracker somehow gets into a state where it has a task in COMMIT_PENDING state that the JobTracker does not know about, the JobTracker will throw NPEs while processing heartbeats. Due to HADOOP-3987, this causes the JT and TT to enter an infinite heartbeat loop with no delays, and the TT fails to make progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5761) JobTracker and TaskTracker enter
infinite loop when TaskTracker reports bad taskid
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712264#action_12712264 ]
Todd Lipcon commented on HADOOP-5761:
-------------------------------------
Lance: can you try updating to 0.19.2 on your cluster and let us know if this problem persists?
> JobTracker and TaskTracker enter infinite loop when TaskTracker reports bad taskid
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-5761
> URL: https://issues.apache.org/jira/browse/HADOOP-5761
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Todd Lipcon
>
> If the TaskTracker somehow gets into a state where it has a task in COMMIT_PENDING state that the JobTracker does not know about, the JobTracker will throw NPEs while processing heartbeats. Due to HADOOP-3987, this causes the JT and TT to enter an infinite heartbeat loop with no delays, and the TT fails to make progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5761) JobTracker and TaskTracker enter
infinite loop when TaskTracker reports bad taskid
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710640#action_12710640 ]
Todd Lipcon commented on HADOOP-5761:
-------------------------------------
I haven't been able to produce this bug (was responding to an email from Lance on core-user), but if you make the assumption that a task can enter such a state, it seems reasonable that the JT should send a Kill action for that task.
To me, this looks similar to HADOOP-5374 but not quite the same. The task probably gets into COMMIT_PENDING state in the same way, but the manifestation was a slightly different stack trace:
{code}
2009-04-30 02:34:40,215 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 54311, call heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@1a93388, false, true, 5341) from 10.253.134.191:42688: error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.mapred.JobTracker.getTasksToSave(JobTracker.java:2130)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:1923)
{code}
(from the original core-user email from Lance)
In the HADOOP-5374 stack trace, the tip variable in getTasksToSave was non-null, and the NPE occured inside shouldCommit. In this case, tip is null.
Lance also reported this being as being on 19.1.
Unfortunately I don't know any way to reproduce this, but it seems logical that if a task somehow enters a bad state (by means some unknown race condition) the JT's interaction with the TT should restore a proper state by killing that task.
> JobTracker and TaskTracker enter infinite loop when TaskTracker reports bad taskid
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-5761
> URL: https://issues.apache.org/jira/browse/HADOOP-5761
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Todd Lipcon
>
> If the TaskTracker somehow gets into a state where it has a task in COMMIT_PENDING state that the JobTracker does not know about, the JobTracker will throw NPEs while processing heartbeats. Due to HADOOP-3987, this causes the JT and TT to enter an infinite heartbeat loop with no delays, and the TT fails to make progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5761) JobTracker and TaskTracker enter
infinite loop when TaskTracker reports bad taskid
Posted by "Lance Riedel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709582#action_12709582 ]
Lance Riedel commented on HADOOP-5761:
--------------------------------------
This is very high priority for us as we have a very unstable production cluster (dying with this issue every day or so).
> JobTracker and TaskTracker enter infinite loop when TaskTracker reports bad taskid
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-5761
> URL: https://issues.apache.org/jira/browse/HADOOP-5761
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Todd Lipcon
>
> If the TaskTracker somehow gets into a state where it has a task in COMMIT_PENDING state that the JobTracker does not know about, the JobTracker will throw NPEs while processing heartbeats. Due to HADOOP-3987, this causes the JT and TT to enter an infinite heartbeat loop with no delays, and the TT fails to make progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5761) JobTracker and TaskTracker enter
infinite loop when TaskTracker reports bad taskid
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709771#action_12709771 ]
Vinod K V commented on HADOOP-5761:
-----------------------------------
Todd and Lance,
Which version of Hadoop are you using? HADOOP-5374 is fixed in 0.19.2 and above, which I think addresses the same issue that you are mentioning. Or is the problem that you are facing different from HADOOP-5374?
> JobTracker and TaskTracker enter infinite loop when TaskTracker reports bad taskid
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-5761
> URL: https://issues.apache.org/jira/browse/HADOOP-5761
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Todd Lipcon
>
> If the TaskTracker somehow gets into a state where it has a task in COMMIT_PENDING state that the JobTracker does not know about, the JobTracker will throw NPEs while processing heartbeats. Due to HADOOP-3987, this causes the JT and TT to enter an infinite heartbeat loop with no delays, and the TT fails to make progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5761) JobTracker and TaskTracker enter
infinite loop when TaskTracker reports bad taskid
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704749#action_12704749 ]
Todd Lipcon commented on HADOOP-5761:
-------------------------------------
In addition to fixing HADOOP-3987 to avoid the infinite loop issue, I think there is a better recovery option here. I would propose something like:
{code:java}
private synchronized List<TaskTrackerAction> getKillActionsForBadTasks(TaskTrackerStatus tts) {
List<TaskStatus> taskStatuses = tts.getTaskReports();
List<TaskTrackerAction> actions = new ArrayList<TaskTrackerAction>();
if (taskStatuses != null) {
for (TaskStatus taskStatus : taskStatuses) {
TaskAttemptID taskId = taskStatus.getTaskID();
if (! taskidToTIPMap.containsKey(taskId)) {
LOG.info("Task Tracker reported status on ID " + taskId + " unknown to JobTracker. Killing task.");
actions.add(new KillTaskAction(taskId));
}
}
}
return actions;
}
{code}
Then fix getTasksToSave (and other instances of the issue) to add "tip != null" checks to their if statements.
Does this sound like a reasonable recovery strategy?
> JobTracker and TaskTracker enter infinite loop when TaskTracker reports bad taskid
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-5761
> URL: https://issues.apache.org/jira/browse/HADOOP-5761
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Todd Lipcon
>
> If the TaskTracker somehow gets into a state where it has a task in COMMIT_PENDING state that the JobTracker does not know about, the JobTracker will throw NPEs while processing heartbeats. Due to HADOOP-3987, this causes the JT and TT to enter an infinite heartbeat loop with no delays, and the TT fails to make progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5761) JobTracker and TaskTracker enter
infinite loop when TaskTracker reports bad taskid
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710694#action_12710694 ]
Amareshwari Sriramadasu commented on HADOOP-5761:
-------------------------------------------------
This should not be an issue after HADOOP-5374 and HADOOP-5247. Both of them got committed to branch 0.19.2
> JobTracker and TaskTracker enter infinite loop when TaskTracker reports bad taskid
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-5761
> URL: https://issues.apache.org/jira/browse/HADOOP-5761
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Todd Lipcon
>
> If the TaskTracker somehow gets into a state where it has a task in COMMIT_PENDING state that the JobTracker does not know about, the JobTracker will throw NPEs while processing heartbeats. Due to HADOOP-3987, this causes the JT and TT to enter an infinite heartbeat loop with no delays, and the TT fails to make progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.