You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2007/05/08 08:25:15 UTC

[jira] Created: (HADOOP-1339) Shuffle should be refactored to a separate task by itself

Shuffle should be refactored to a separate task by itself
---------------------------------------------------------

                 Key: HADOOP-1339
                 URL: https://issues.apache.org/jira/browse/HADOOP-1339
             Project: Hadoop
          Issue Type: New Feature
          Components: mapred
            Reporter: Devaraj Das
             Fix For: 0.14.0


Currently, shuffle phase is part of the reduce task. The idea here is to move out the shuffle as a first-class task. This will improve the usage of the network since we will then be able to schedule shuffle tasks independently, and later on pin reduce tasks to those nodes. This will make most sense for apps where there are multiple waves of reduces (the second wave of reduces can directly start off doing the "reducer" phase).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.