[jira] Commented: (HADOOP-1338) Improve the shuffle phase by using the "connection: keep-alive" and doing batch transfers of files

["Runping Qi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648618#action_12648618 ] 

Runping Qi commented on HADOOP-1338:
------------------------------------



Had an offline chat with Devaraj about this issue.
Here is a proposal based on the chat.


When a reducer need to fetch map outputs from a TT, it sends a request to the TT with a list of segments, not just one, it needs from the TT
Upon receiving the request, the TT builds a response consisting of a subset of the requested map output segments, up to certain upper limit on the 
total size. TT can do so with a sngle pass of sequential reads of the map output file, instead of multiple random reads.
This will significantly lower the number of round trips between the reducer and the TT and the number of random reads the TT needs to do for 
the jobs with large number of mappers and the with a lot of small map output segments.

> Improve the shuffle phase by using the "connection: keep-alive" and doing batch transfers of files
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1338
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1338
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>
> We should do transfers of map outputs at the granularity of  *total-bytes-transferred* rather than the current way of transferring a single file and then closing the connection to the server. A single TaskTracker might have a couple of map output files for a given reduce, and we should transfer multiple of them (upto a certain total size) in a single connection to the TaskTracker. Using HTTP-1.1's keep-alive connection would help since it would keep the connection open for more than one file transfer. We should limit the transfers to a certain size so that we don't hold up a jetty thread indefinitely (and cause timeouts for other clients).
> Overall, this should give us improved performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.