[jira] Commented: (HADOOP-3136) Assign multiple tasks per TaskTracker heartbeat

["Joydeep Sen Sarma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618922#action_12618922 ] 

Joydeep Sen Sarma commented on HADOOP-3136:
-------------------------------------------

i don't know how big a change we are envisaging here. my suggestion was thinking if we wanted to minimize changes to current code base. I don't think the protocol suggested would send any more heartbeats than we do in the worst case today (or that at least it could be tuned to get that property) - but if the plan going forward is to reduce heartbeats - then that's not good enough.

If we are planning non-trivial changes - it seems to me that it would be nice as a first step to have scheduling decisions dispatched pro-actively to slaves on new job arrival (after considering global resource availability). that itself should save on latency of scheduling on an idling cluster. on a backloaded cluster, scheduling reactively to heartbeats seems reasonable (in fact - on a backloaded cluster - it's likely that each heartbeat would only advertize one available slot. unless of course we move away from current protocol for sending heartbeat on task completion).

the other significant improvement i can think of is maintaining a runnable task queue on each TT (in addition to the ones running). that way there would be no idle time lost to getting new tasks. the protocol of dispatching tasks to TT runnable queues could happen in the background where one could make optimizations to reduce network traffic. (for example - if the TT had a good sized queue of runnable tasks then it could afford to delay sending heartbeats on task completion (since delaying would no longer cause idle time)). This may also allow the JT to make batched scheduling decisions (instead of immediately looking for something to schedule when slots become free - the JT could wait and accumulate some larger number of free slots before making a global scheduling decision over all those slots and the available tasks. this might make things more globally optimal while still not causing idle time (since there every TT has a queue of runnable tasks)).

> Assign multiple tasks per TaskTracker heartbeat
> -----------------------------------------------
>
>                 Key: HADOOP-3136
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3136
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Arun C Murthy
>             Fix For: 0.19.0
>
>
> In today's logic of finding a new task, we assign only one task per heartbeat.
> We probably could give the tasktracker multiple tasks subject to the max number of free slots it has - for maps we could assign it data local tasks. We could probably run some logic to decide what to give it if we run out of data local tasks (e.g., tasks from overloaded racks, tasks that have least locality, etc.). In addition to maps, if it has reduce slots free, we could give it reduce task(s) as well. Again for reduces we could probably run some logic to give more tasks to nodes that are closer to nodes running most maps (assuming data generated is proportional to the number of maps). For e.g., if rack1 has 70% of the input splits, and we know that most maps are data/rack local, we try to schedule ~70% of the reducers there.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.