You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2008/05/09 12:38:55 UTC
[jira] Created: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
failed tasks may stay forever in TaskTracker.runningJobs
--------------------------------------------------------
Key: HADOOP-3370
URL: https://issues.apache.org/jira/browse/HADOOP-3370
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Zheng Shao
Priority: Critical
The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
Details:
TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
which calls TaskTracker.TaskInProgress.taskFinished,
which calls TaskTracker.TaskInProgress.cleanup(),
which calls TaskTracker.tasks.remove(taskId).
In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
Then the failure is reported to JobTracker.
JobTracker.heartbeat will call processHeartbeat,
which calls updateTaskStatuses,
which calls tip.getJob().updateTaskStatus,
which calls JobInProgress.failedTask,
which calls JobTracker.markCompletedTaskAttempt,
which puts the task to trackerToMarkedTasksMap,
and then JobTracker.heartbeat will call removeMarkedTasks,
which call removeTaskEntry,
which removes it from trackerToTaskMap.
JobTracker.heartbeat will also call JobTracker.getTasksToKill,
which reads from trackerToTaskMap for <tracker, task> pairs,
and ask tracker to KILL the task or job of the task.
In the case there is only one task for a specific job on a specific tracker
and that task failed (NOTE: and that task is not the last failed try of the
job - otherwise JobTracker.getTasksToKill will pick it up before
removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
tracker will not receive the KILL task or KILL job message from the JobTracker.
As a result, the task will remain in TaskTracker.runningJobs forever.
Solution:
Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596650#action_12596650 ]
Hadoop QA commented on HADOOP-3370:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12382012/3370-2.patch
against trunk revision 656122.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2460/console
This message is automatically generated.
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Status: In Progress (was: Patch Available)
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596699#action_12596699 ]
Amareshwari Sriramadasu commented on HADOOP-3370:
-------------------------------------------------
Looks like the failure is due logger intialization warnings.
<noformat>
-------------------- DEBUG OUT---------------------
Test Script
Bailing out
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.TaskRunner).
log4j:WARN Please initialize the log4j system properly.
<noformat>
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-3370:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Zheng!
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Attachment: 3370-1.patch
Simple fix. I also included code to show the content of runningTasks on http interface.
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Zheng Shao
> Priority: Critical
> Attachments: 3370-1.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595686#action_12595686 ]
Zheng Shao commented on HADOOP-3370:
------------------------------------
Details about a potential solution:
1. On failed task, remove the task from runningJobs, but do not delete runningJobs job entry even if it's the only task of the job; (which means we should NOT call TaskTracker.removeTaskFromJob)
2. JobTracker should keep another data structure: jobsToTracker, for recording all the TaskTrackers that a job has started a task on.
3. When the job finished, JobTracker will send "KILL" job command to the TaskTrackers, based on jobsToTracker data structure.
An alternative:
On failed task, remove the task from runningJobs, AND if it's the only task of the job, remove the job directory (which means we should call TaskTracker.removeTaskFromJob, PLUS delete the job directory)
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Zheng Shao
> Priority: Critical
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Attachment: 3370-3.patch
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Status: In Progress (was: Patch Available)
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595778#action_12595778 ]
Hadoop QA commented on HADOOP-3370:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381804/3370-1.patch
against trunk revision 654973.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2444/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2444/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2444/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2444/console
This message is automatically generated.
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Zheng Shao
> Priority: Critical
> Attachments: 3370-1.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596689#action_12596689 ]
Zheng Shao commented on HADOOP-3370:
------------------------------------
can somebody help me restart the hudson test? It seems there are some transient errors.
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596684#action_12596684 ]
Hadoop QA commented on HADOOP-3370:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12382014/3370-3.patch
against trunk revision 656122.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2462/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2462/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2462/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2462/console
This message is automatically generated.
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Attachment: 3370-2.patch
1. removed commented code;
2. removed extra "this." where not necessary.
I will put that "KILLJOB" thing in a separate issue - which helps cleaning out the local dirs of the job in corner cases.
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596691#action_12596691 ]
Arun C Murthy commented on HADOOP-3370:
---------------------------------------
Looks like org.apache.hadoop.mapred.TestMiniMRMapRedDebugScript.testMapDebugScript failed:
{noformat}
junit.framework.ComparisonFailure: expected:<...> but was:<...
{noformat}
Looks like it might have broken the feature where u can added a debug-script for your map/reduce tasks, look at TestMiniMRMapRedDebugScript for an example.
Does it succeed on your machine?
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596653#action_12596653 ]
dhruba borthakur commented on HADOOP-3370:
------------------------------------------
Thanks Arun for reviewing this one. Really appreciate it.
I think the HadoopQA pach process had some problem in applying and merging this patch with trunk. Maybe I will cancel and re-submit this issue to re-trigger another Hudson test.
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao reassigned HADOOP-3370:
----------------------------------
Assignee: Zheng Shao
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Attachments: 3370-1.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Attachment: 3370-3.patch
my fault. regenerated the patch.
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611932#action_12611932 ]
Amareshwari Sriramadasu commented on HADOOP-3370:
-------------------------------------------------
All the tests passed on branch 0.17, on my machine
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch, patch-3370-0.17.txt
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Status: Patch Available (was: In Progress)
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Status: Patch Available (was: Open)
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Zheng Shao
> Priority: Critical
> Attachments: 3370-1.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Status: Patch Available (was: In Progress)
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-3370:
----------------------------------
Fix Version/s: 0.18.0
Affects Version/s: 0.17.0
Status: Open (was: Patch Available)
Zheng, apologies for being late to get to this - couple of comments:
1. Please do not comment out code which is no longer required, just delete it.
2. HADOOP-3297 changed the way we get TaskCompletionEvents, it is no longer once in 5s. Just FYI.
3. If you don't mind, please do not use this.<func> whenever calling <func> suffices.
4. As you mentioned, the other option is to send a KillJobAction for all trackers on which tasks ran at the end of the job. This is a really useful feature and would make me very happy if you took that route! *smile* - however, I won't hold it against this patch; we could do it as a separate issue.
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-3370:
--------------------------------------------
Attachment: patch-3370-0.17.txt
Patch for 0.17
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch, patch-3370-0.17.txt
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Attachment: 3370-4.patch
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3370) failed tasks may stay
forever in TaskTracker.runningJobs
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596699#action_12596699 ]
amareshwari edited comment on HADOOP-3370 at 5/14/08 2:56 AM:
--------------------------------------------------------------------------
Looks like the failure is due logger intialization warnings.
{noformat}
-------------------- DEBUG OUT---------------------
Test Script
Bailing out
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.TaskRunner).
log4j:WARN Please initialize the log4j system properly.
{noformat}
was (Author: amareshwari):
Looks like the failure is due logger intialization warnings.
<noformat>
-------------------- DEBUG OUT---------------------
Test Script
Bailing out
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.TaskRunner).
log4j:WARN Please initialize the log4j system properly.
<noformat>
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596802#action_12596802 ]
Hadoop QA commented on HADOOP-3370:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12382024/3370-4.patch
against trunk revision 656270.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2466/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2466/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2466/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2466/console
This message is automatically generated.
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-3370:
----------------------------------
Fix Version/s: (was: 0.18.0)
0.17.2
I merged this into branch-0.17 also.
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.17.2
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch, patch-3370-0.17.txt
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Attachment: (was: 3370-3.patch)
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HADOOP-3370:
-------------------------------
Status: Patch Available (was: Open)
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: 3370-1.patch, 3370-2.patch
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3370) failed tasks may stay forever in
TaskTracker.runningJobs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624681#action_12624681 ]
Hudson commented on HADOOP-3370:
--------------------------------
Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])
> failed tasks may stay forever in TaskTracker.runningJobs
> --------------------------------------------------------
>
> Key: HADOOP-3370
> URL: https://issues.apache.org/jira/browse/HADOOP-3370
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Critical
> Fix For: 0.17.2
>
> Attachments: 3370-1.patch, 3370-2.patch, 3370-3.patch, 3370-4.patch, patch-3370-0.17.txt
>
>
> The net effect of this is that, with a long-running TaskTracker, it takes long long time for ReduceTasks on that TaskTracker to fetch map outputs - TaskTracker does that for all reduce tasks in TaskTracker .runningJobs, including those stale ReduceTasks. There is a 5-second delay between 2 requests, which makes it a long time for a running reducetask to get the map output locations, when there are tens of stale ReduceTasks. Of course this also blows up the memory but that is not a too big problem at its rate.
> I've verified the bug by adding an html table for TaskTracker.runningJobs on TaskTracker http interface, on a 2-node machine, with a single mapper single reducer job, in which mapper succeeds and reducer fails. I can still see the ReduceTask in TaskTracker.runningJobs, while it's not in the first 2 tables (TaskTracker.tasks and TaskTracker.runningTasks).
> Details:
> TaskRunner.run() will call TaskTracker.reportTaskFinished() when the task fails,
> which calls TaskTracker.TaskInProgress.taskFinished,
> which calls TaskTracker.TaskInProgress.cleanup(),
> which calls TaskTracker.tasks.remove(taskId).
> In short, it remove a failed task from TaskTracker.tasks, but not TaskTracker.runningJobs.
> Then the failure is reported to JobTracker.
> JobTracker.heartbeat will call processHeartbeat,
> which calls updateTaskStatuses,
> which calls tip.getJob().updateTaskStatus,
> which calls JobInProgress.failedTask,
> which calls JobTracker.markCompletedTaskAttempt,
> which puts the task to trackerToMarkedTasksMap,
> and then JobTracker.heartbeat will call removeMarkedTasks,
> which call removeTaskEntry,
> which removes it from trackerToTaskMap.
> JobTracker.heartbeat will also call JobTracker.getTasksToKill,
> which reads from trackerToTaskMap for <tracker, task> pairs,
> and ask tracker to KILL the task or job of the task.
> In the case there is only one task for a specific job on a specific tracker
> and that task failed (NOTE: and that task is not the last failed try of the
> job - otherwise JobTracker.getTasksToKill will pick it up before
> removeMarkedTasks comes in and remove it from trackerToTaskMap), the task
> tracker will not receive the KILL task or KILL job message from the JobTracker.
> As a result, the task will remain in TaskTracker.runningJobs forever.
> Solution:
> Remove the task from TaskTracker.runningJobs at the same time when we remove it from TaskTracker.tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.