[jira] Commented: (HADOOP-1339) Shuffle should be refactored to a separate task by itself

["Amar Kamat (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12574190#action_12574190 ] 

Amar Kamat commented on HADOOP-1339:
------------------------------------

Does it makes sense to spawn a thread from the task tracker rather than a separate jvm? The reason being that the shuffle code is again a framework code. 

> Shuffle should be refactored to a separate task by itself
> ---------------------------------------------------------
>
>                 Key: HADOOP-1339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1339
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Devaraj Das
>
> Currently, shuffle phase is part of the reduce task. The idea here is to move out the shuffle as a first-class task. This will improve the usage of the network since we will then be able to schedule shuffle tasks independently, and later on pin reduce tasks to those nodes. This will make most sense for apps where there are multiple waves of reduces (the second wave of reduces can directly start off doing the "reducer" phase).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.