You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by Konstantin Shvachko <sh...@yahoo-inc.com> on 2006/04/14 04:20:32 UTC

Hadoop performance test

Here are some Hadoop performance numbers.
Cluster size is ~400 nodes.
Using TestDFSIO (see HADOOP-95) I run N=1,2,4,...,512 maps.
Each map either write 320 Mb into a file or reads the same amount.
The reducer collects statistics.
Finally the following info is appended to a local file

 *    read or write test
 *    date and time the test finished  
 *    number of files
 *    total number of bytes processed
 *    throughput in mb/sec (total number of bytes / sum of processing times)
 *    average IO rate in mb/sec per file
 *    standard IO rate deviation

The resulting file is attached.

--Konstantin