Hadoop 0.20.203 vertical scalability

[blah blah <tm...@gmail.com>]

Hi

I am using Hadoop 0.20.203.

I have performed simple vertical scalability experiments of Hadoop with the
use of Graph datasets and BFS algorithm. My experiment configuration is
20workers + Master. In each test I divided the Map slots and Reduce Slots
equally (M==R), I can process the Job(single BFS step) in a single wave.
For the experiments I have used the wall clock time metric. The total size
of the dataset is around 311mb. Below are my recorded times (I'll ask the
question after presenting them).

Per worker : 1CPU (1Slot)  2CPU(2Slots) 3CPU  4CPU 5CPU  6CPU 7CPU  // 1
cpu is always left for deamons
Time          : 429s              434s              417s   412s   429s
  430s   470s   // presented times are averages from 10 algorithm executions

As it can be seen initial trend (1->2 cpu) is actually increasing the
execution time. However another trend occurs for the (2->3 cpu), which
lasts till 4cpu (time decrease). After that the execution time increase all
the way till 7cpus. Can some one explain to me why Hadoop scales like that,
I can understand that for the large number of CPUs, tasks compete over BUS,
local disk I/O. But I can not explain (do not understand) the change in
trends.

Thanks in advance.
regards
blah