how dfs.write.packet.size impact write through put of HDFS, strange result?

[elton sky <el...@gmail.com>]

Hello everyone,

I am doing some evaluation on my 6 nodes mini cluster. Each node has 4 core
Intel(R) Xeon(R) CPU 5130  @ 2.00GHz, 8GB memory, 500 GB disk, running Linux
version 2.6.18-164.11.1.el5(Red Hat 4.1.2-46).

I was trying to use different packet size (dfs.write.packet.size) and
bytePerChunksum(bpc) (io.bytes.per.checksum), with default block size and
buffer size, to see how they impact the performance of creating a 1GB file.

The default packet size is 64K, and, surprisingly, if I increase this value
to 256K, 1M, etc, the performance drops gradually. Below is some result I
got:

time(ms): 25742  fileSize:1073741824 block size:67108864 bufferSize: 4096
bpc: 1024 packet size: 1048576

time(ms): 24960  fileSize:1073741824 block size:67108864 bufferSize: 4096
bpc: 8192 packet size: 1048576

time(ms): 25412  fileSize:1073741824 block size:67108864 bufferSize: 4096
bpc: 65536 packet size: 1048576

time(ms): 20322   fileSize:1073741824 block size:67108864 bufferSize: 4096
bpc: 1024 packet size: 262144

time(ms): 19048  fileSize:1073741824 block size:67108864 bufferSize: 4096
bpc: 8192 packet size: 262144

time(ms): 19227  fileSize:1073741824 block size:67108864 bufferSize: 4096
bpc: 65536 packet size: 262144

time(ms): 15141   fileSize:1073741824 block size:67108864 bufferSize: 4096
bpc: 1024 packet size: 65536

time(ms): 14983  fileSize:1073741824 block size:67108864 bufferSize: 4096
bpc: 8192 packet size: 65536

time(ms): 14801  fileSize:1073741824 block size:67108864 bufferSize: 4096
bpc: 65536 packet size: 65536

...

>From these results I can see performance declined with packet size
increases. bpc didn't affect much though.

>From my understanding, I think if I use less packets then I introduces less
overhead and less read/write roundtrip. I couldn't explain what happened
here.

Can anyone give me some idea here?