You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Nikola Vujic (JIRA)" <ji...@apache.org> on 2014/04/04 15:17:16 UTC
[jira] [Commented] (MAPREDUCE-5791) Shuffle phase is slow in
Windows - FadviseFileRegion::transferTo does not read disks efficiently
[ https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13959934#comment-13959934 ]
Nikola Vujic commented on MAPREDUCE-5791:
-----------------------------------------
In order to check the performance impact of this patch, I have tested the patch on a 80 node hadoop cluster on Windows. Here are the results:
Terasort 5 TB
1540 map tasks
770 reduce tasks
Elapsed Avg Map Time Avg Reduce time Avg Shuffle Time Avg Merge Time
default 3194.67 402.33 248.67 1471.33 12.33
optimized shuffle 2411.00 392.00 689.67 674.67 17.17
default/optmized 1.33 1.03 0.36 2.18 0.72
* optimized shuffle is configured to use 512K buffer size for the buffer copy shuffle.
** presented numbers are avg of at least 3 runs.
Optimized shuffle version is 1.33x faster than the default version. Gain in the shuffle phase alone is 2.18x.
> Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
> ------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5791
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 3.0.0, 2.3.0
> Reporter: Nikola Vujic
> Assignee: Nikola Vujic
> Fix For: 3.0.0, 2.4.0
>
> Attachments: MAPREDUCE-5791.patch, MAPREDUCE-5791.patch, MAPREDUCE-5791.patch
>
>
> transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using transferTo method from a FileChannel to transfer data from a disk to socket. This is performing slow in Windows, slower than in Linux. The reason is that transferTo method for the java.nio is issuing 32K IO requests all the time. In Windows, these 32K transfers are not optimal and we don't get the best performance form the underlying IO subsystem. In order to achieve better performance when reading from the drives, we need to read data in bigger chunks, 512K for example.
--
This message was sent by Atlassian JIRA
(v6.2#6252)