You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2006/05/25 18:59:30 UTC

[jira] Updated: (HADOOP-254) use http to shuffle data between the maps and the reduces

     [ http://issues.apache.org/jira/browse/HADOOP-254?page=all ]

Owen O'Malley updated HADOOP-254:
---------------------------------

    Attachment: http-shuffle.patch

This patch introduces the use of http to shuffle the data from maps to reduces. Using this patch, the shuffle mostly keeps up with the maps during my sort benchmark.

This patch also allows the task tracker's http servers to try multiple ports until it finds one that works.

> use http to shuffle data between the maps and the reduces
> ---------------------------------------------------------
>
>          Key: HADOOP-254
>          URL: http://issues.apache.org/jira/browse/HADOOP-254
>      Project: Hadoop
>         Type: Improvement

>   Components: mapred
>     Versions: 0.2.1
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>      Fix For: 0.3
>  Attachments: http-shuffle.patch
>
> To speed up the shuffle time, I'll use http (via the task tracker's jetty server) to send the map outputs.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira