You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Jian He (JIRA)" <ji...@apache.org> on 2015/05/18 06:12:00 UTC
[jira] [Updated] (HADOOP-7) MapReduce has a series of problems
concerning task-allocation to worker nodes
[ https://issues.apache.org/jira/browse/HADOOP-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jian He updated HADOOP-7:
-------------------------
Assignee: (was: Jian He)
> MapReduce has a series of problems concerning task-allocation to worker nodes
> -----------------------------------------------------------------------------
>
> Key: HADOOP-7
> URL: https://issues.apache.org/jira/browse/HADOOP-7
> Project: Hadoop Common
> Issue Type: Bug
> Environment: All
> Reporter: Mike Cafarella
> Fix For: 0.1.0
>
> Attachments: jobtracker.patch
>
>
> The MapReduce JobTracker is not great at allocating tasks to TaskTracker worker nodes.
> Here are the problems:
> 1) There is no speculative execution of tasks
> 2) Reduce tasks must wait until all map tasks are completed before doing any work
> 3) TaskTrackers don't distinguish between Map and Reduce jobs. Also, the number of
> tasks at a single node is limited to some constant. That means you can get weird deadlock
> problems upon machine failure. The reduces take up all the available execution slots, but they
> don't do productive work, because they're waiting for a map task to complete. Of course, that
> map task won't even be started until the reduce tasks finish, so you can see the problem...
> 4) The JobTracker is so complicated that it's hard to fix any of these.
> The right solution is a rewrite of the JobTracker to be a lot more flexible in task handling.
> It has to be a lot simpler. One way to make it simpler is to add an abstraction I'll call
> "TaskInProgress". Jobs are broken into chunks called TasksInProgress. All the TaskInProgress
> objects must be complete, somehow, before the Job is complete.
> A single TaskInProgress can be executed by one or more Tasks. TaskTrackers are assigned Tasks.
> If a Task fails, we report it back to the JobTracker, where the TaskInProgress lives. The TIP can then
> decide whether to launch additional Tasks or not.
> Speculative execution is handled within the TIP. It simply launches multiple Tasks in parallel. The
> TaskTrackers have no idea that these Tasks are actually doing the same chunk of work. The TIP
> is complete when any one of its Tasks are complete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)