io performance

[Gang Luo <lg...@yahoo.com.cn>]

Hi all,
I got some questions about the performance of IO in hadoop. What I want to know is which of the three phases (mapper input, shuffle and reducer output) is the bottleneck.

First, compared with the shuffle phase (mapper output, data transfer over network and reducer write it to local disk), does it cost more time to read the input source to mappers? My concern is that to read a large table also need to transfer different splits to different mappers over network. Is hadoop smart enough to make input consume less time than shuffle?

Second, for the final result will be stored in HDFS, we need to duplicate several copies (by default, 3 copies). This need to transfer more than one copy of the result over network. If we run another mapreduce job which is dependent on this result, I think the latency for replicating the result is non-trivial, right? What is the performance compared with mapper input and shuffle?

Thanks.

 
Gang Luo
---------
Department of Computer Science
Duke University
(919)316-0993
gang.luo@duke.edu


      ___________________________________________________________ 
  好玩贺卡等你发，邮箱贺卡全新上线！ 
http://card.mail.cn.yahoo.com/

Re: io performance

[brien colwell <xc...@gmail.com>]

May be of help ... In my experience there is not a single bottleneck.
Even your tuple representation may disproportionally impact performance.
Too-granular tuples resulting in redundant values will slow down the
shuffle, which does at least 2 serialize/de-serialize ops per tuple.
Benchmarks on my internal cluster show HDFS seek times around 25ms,
higher than a typical physical disk. Hence I'm under the impression the
number of input/output files affects IO performance more than on a
typical physical disk.

A paper from Berkeley EECS about pipelining between Reduce and Map may
be of interest.
http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-136.html






Gang Luo wrote:
> Hi all,
> I got some questions about the performance of IO in hadoop. What I want to know is which of the three phases (mapper input, shuffle and reducer output) is the bottleneck.
>
> First, compared with the shuffle phase (mapper output, data transfer over network and reducer write it to local disk), does it cost more time to read the input source to mappers? My concern is that to read a large table also need to transfer different splits to different mappers over network. Is hadoop smart enough to make input consume less time than shuffle?
>
> Second, for the final result will be stored in HDFS, we need to duplicate several copies (by default, 3 copies). This need to transfer more than one copy of the result over network. If we run another mapreduce job which is dependent on this result, I think the latency for replicating the result is non-trivial, right? What is the performance compared with mapper input and shuffle?
>
> Thanks.
>
>  
> Gang Luo
> ---------
> Department of Computer Science
> Duke University
> (919)316-0993
> gang.luo@duke.edu
>
>
>       ___________________________________________________________ 
>   好玩贺卡等你发，邮箱贺卡全新上线！ 
> http://card.mail.cn.yahoo.com/
>