You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Min Zhou <co...@gmail.com> on 2010/12/17 05:45:14 UTC
Questions on HADOOP-6713
Hi all,
After apply the patch from HADOOP-6713 into our hadoop source tree, I
did a benchmark on multi-readers RPC, and didn't found any throughput
improvement .
The attachment of this mail is a patch of my rpc benmark. Test
scenario listed below.
Network: Gigabit LAN
CPU: 8 core Intel(R) Xeon(R) CPU E5420 @ 2.50GHz
Memory: 8GB
================== scenario 1 =======================
server side:
# export HADOOP_OPS="-Xmx4096m -Xms256m"
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
-handlers 1 -readers 10 -durationPerRPC 100
client side:
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
-host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
10/12/17 12:25:07 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
10/12/17 12:26:02 INFO ipc.RPCBenchmark: # operations: 100
10/12/17 12:26:02 INFO ipc.RPCBenchmark: Elapsed Time: 55073 ms
10/12/17 12:26:02 INFO ipc.RPCBenchmark: Ops per sec: 1.815771793800955
10/12/17 12:26:02 INFO ipc.RPCBenchmark: Average Time: 5249 ms
================== scenario 2 =======================
server side:
# export HADOOP_OPS="-Xmx4096m -Xms256m"
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
-handlers 10 -readers 10 -durationPerRPC 100
client side:
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
-host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
10/12/17 12:28:23 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
10/12/17 12:29:11 INFO ipc.RPCBenchmark: # operations: 100
10/12/17 12:29:11 INFO ipc.RPCBenchmark: Elapsed Time: 47784 ms
10/12/17 12:29:11 INFO ipc.RPCBenchmark: Ops per sec: 2.092750711535242
10/12/17 12:29:11 INFO ipc.RPCBenchmark: Average Time: 4581 ms
================== scenario 3 =======================
server side:
# export HADOOP_OPS="-Xmx4096m -Xms256m"
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
-handlers 10 -readers 1 -durationPerRPC 100
client side:
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
-host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
10/12/17 12:31:19 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
10/12/17 12:32:08 INFO ipc.RPCBenchmark: # operations: 100
10/12/17 12:32:08 INFO ipc.RPCBenchmark: Elapsed Time: 49155 ms
10/12/17 12:32:08 INFO ipc.RPCBenchmark: Ops per sec: 2.034381039568711
10/12/17 12:32:08 INFO ipc.RPCBenchmark: Average Time: 4722 ms
================== scenario 4 =======================
server side:
# export HADOOP_OPS="-Xmx4096m -Xms256m
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
-handlers 1 -readers 1 -durationPerRPC 100
client side:
# hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
-host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
10/12/17 12:36:04 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
10/12/17 12:36:59 INFO ipc.RPCBenchmark: # operations: 100
10/12/17 12:36:59 INFO ipc.RPCBenchmark: Elapsed Time: 55054 ms
10/12/17 12:36:59 INFO ipc.RPCBenchmark: Ops per sec: 1.816398445162931
10/12/17 12:36:59 INFO ipc.RPCBenchmark: Average Time: 5278 ms
Statistics above indicates that a single reader can consume 200,000
8-bytes Strings very fast. Ten concurrent readers should consume the
same data more quickly, but reader isn't the bottleneck of RPC at
least in these scenarios. Can you give an example which scenario
HADOOP-6713 will take affect?
Thanks,
Min
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
Re: Questions on HADOOP-6713
Posted by Min Zhou <co...@gmail.com>.
Anybody can help me with this?
On Fri, Dec 17, 2010 at 12:45 PM, Min Zhou <co...@gmail.com> wrote:
> Hi all,
>
> After apply the patch from HADOOP-6713 into our hadoop source tree, I
> did a benchmark on multi-readers RPC, and didn't found any throughput
> improvement .
> The attachment of this mail is a patch of my rpc benmark. Test
> scenario listed below.
>
> Network: Gigabit LAN
> CPU: 8 core Intel(R) Xeon(R) CPU E5420 @ 2.50GHz
> Memory: 8GB
>
> ================== scenario 1 =======================
> server side:
> # export HADOOP_OPS="-Xmx4096m -Xms256m"
> # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
> -handlers 1 -readers 10 -durationPerRPC 100
>
> client side:
> # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
> -host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
> 10/12/17 12:25:07 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
> 10/12/17 12:26:02 INFO ipc.RPCBenchmark: # operations: 100
> 10/12/17 12:26:02 INFO ipc.RPCBenchmark: Elapsed Time: 55073 ms
> 10/12/17 12:26:02 INFO ipc.RPCBenchmark: Ops per sec: 1.815771793800955
> 10/12/17 12:26:02 INFO ipc.RPCBenchmark: Average Time: 5249 ms
>
> ================== scenario 2 =======================
> server side:
> # export HADOOP_OPS="-Xmx4096m -Xms256m"
> # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
> -handlers 10 -readers 10 -durationPerRPC 100
>
> client side:
> # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
> -host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
> 10/12/17 12:28:23 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
> 10/12/17 12:29:11 INFO ipc.RPCBenchmark: # operations: 100
> 10/12/17 12:29:11 INFO ipc.RPCBenchmark: Elapsed Time: 47784 ms
> 10/12/17 12:29:11 INFO ipc.RPCBenchmark: Ops per sec: 2.092750711535242
> 10/12/17 12:29:11 INFO ipc.RPCBenchmark: Average Time: 4581 ms
>
> ================== scenario 3 =======================
> server side:
> # export HADOOP_OPS="-Xmx4096m -Xms256m"
> # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
> -handlers 10 -readers 1 -durationPerRPC 100
>
> client side:
> # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
> -host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
> 10/12/17 12:31:19 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
> 10/12/17 12:32:08 INFO ipc.RPCBenchmark: # operations: 100
> 10/12/17 12:32:08 INFO ipc.RPCBenchmark: Elapsed Time: 49155 ms
> 10/12/17 12:32:08 INFO ipc.RPCBenchmark: Ops per sec: 2.034381039568711
> 10/12/17 12:32:08 INFO ipc.RPCBenchmark: Average Time: 4722 ms
>
> ================== scenario 4 =======================
> server side:
> # export HADOOP_OPS="-Xmx4096m -Xms256m
> # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server
> -handlers 1 -readers 1 -durationPerRPC 100
>
> client side:
> # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client
> -host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000
> 10/12/17 12:36:04 INFO ipc.RPCBenchmark: Starting 100 rpc call(s).
> 10/12/17 12:36:59 INFO ipc.RPCBenchmark: # operations: 100
> 10/12/17 12:36:59 INFO ipc.RPCBenchmark: Elapsed Time: 55054 ms
> 10/12/17 12:36:59 INFO ipc.RPCBenchmark: Ops per sec: 1.816398445162931
> 10/12/17 12:36:59 INFO ipc.RPCBenchmark: Average Time: 5278 ms
>
> Statistics above indicates that a single reader can consume 200,000
> 8-bytes Strings very fast. Ten concurrent readers should consume the
> same data more quickly, but reader isn't the bottleneck of RPC at
> least in these scenarios. Can you give an example which scenario
> HADOOP-6713 will take affect?
>
> Thanks,
> Min
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com