You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Gokulakannan M <go...@huawei.com> on 2010/04/12 16:21:41 UTC
response time increases lineraly for each thread!
Hi,
When I tested the performance of a 11 node hadoop
cluster(private nw) for the write scenario with several threads, I noticed
that the time for each write increases linearly(what ever the file size may
be).
Actually I'm running these threads as a single process in the
system where namenode is running. I'm getting inconsistent results when I
run this suite each time. The results are fluctuating. What might be the
bottlenecks?
I think running this in parallel by spawning multiple shells
will give consistent result. But running these in the same system will give
exact results??
Any ways to measure the performance correctly?
Appreciated any help in this.
Thanks,
Gokul
RE: response time increases lineraly for each thread!
Posted by Sagar Shukla <sa...@persistent.co.in>.
Hi Gokul,
Just to add to what Todd mentioned, in Hadoop - though you are uploading the file on local disk but it would get distributed across multiple nodes maintaining the replication factor and hence will generate network traffic. So this can affect performance as time goes by.
Thanks,
Sagar
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Wednesday, April 14, 2010 2:04 AM
To: gokulm@huawei.com
Cc: hdfs-user@hadoop.apache.org; Sagar Shukla
Subject: Re: response time increases lineraly for each thread!
Hi Gokul,
This linear degradation makes sense. If you're saturating your outgoing network bandwidth from the node that's doing the writing, then doubling the number of writers will decrease the amount of bandwidth available to each writer by a factor of two. So of course each writer will take twice as long if there are twice as many writers.
This is true regardless of whether you are disk bound or network bound on the local node - you have a finite resource (throughput) which you're splitting among more clients.
If you run the concurrent writers from different nodes you'll probably see that the write time stays constant until you start to saturate bandwidth on all of the nodes.
Does this make sense?
-Todd
On Tue, Apr 13, 2010 at 2:15 AM, Gokulakannan M <go...@huawei.com>> wrote:
Todd,
I am benchmarking the performance for the write scenario(Not only for single user but for concurrent users). I've implemented a simple FileWrite() method that takes namenode address and source file and write into hdfs. I'm measuring the response time by just getting the time diff before and after calling the FileWrite() method (by using System.currentMillliSeconds()).I've created several threads for this FileWrite() method and measure the performance. Is anything wrong with the test code?
No I didn't configure dfs.data.dir to all volumes but the current one. So the issue might be due to disk latency? The total space available is 750GB(seen from name UI).
Thanks,
Gokul
[cid:image001.gif@01CADBB8.398A97D0]
________________________________
From: Todd Lipcon [mailto:todd@cloudera.com<ma...@cloudera.com>]
Sent: Tuesday, April 13, 2010 10:54 AM
To: gokulm@huawei.com<ma...@huawei.com>; hdfs-user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: sagar_shukla@persistent.co.in<ma...@persistent.co.in>
Subject: Re: response time increases lineraly for each thread!
Hi Gokul,
Would it be possible to post your benchmark code somewhere? I'm not clear on what exact thing you're measuring - IOUtils.copyBytes is a pretty general method.
Also, what disk hardware are you using? Have you configured dfs.data.dir to point to all local drives on each DN? Could very well be a factor of disk seeks limiting you.
-Todd
On Mon, Apr 12, 2010 at 10:18 PM, Gokulakannan M <go...@huawei.com>> wrote:
@Todd,
I am using 1 namenode and 10 datanodes. Replication factor is 3.
These are the details about my test suite. It uses the simple write code that uses IOUtils.copyBytes() and calculates the response time for each write for a given number of users(threads).
From namenode, I am running the test suite which calculates the response time for writing a file for several concurrent users(1 , 4 , 9 , 20 ). Whatever the file size may be(I tried with 100 MB , 1 GB and 5 GB files), I find that the response time increases linearly.
These are the results for a 100 MB file.
No of Threads
Response time(milli sec)
1
2199
4
2763
2774
34766
64488
9
3620
4897
5018
9991
65501
124156
183639
243631
314233
20
6702
6784
6985
8987
9202
9271
9925
10752
68237
70878
73469
75261
77507
82061
129942
137098
141822
194199
269353
322328
You can see the write time increases for each thread.
@ Sagar:
I think network is not the issue here. This is a private network with high bandwidth. Moreover I tried the running the same test suite with several configurations(pseudo dist mode, 1 NN 1 DN Cluster , 1 NN 3 DN cluster) in my usual network(typically with low bandwidth) and obtained the results like above :(
Thanks,
Gokul
[cid:image001.gif@01CADBB8.398A97D0]
________________________________
From: Todd Lipcon [mailto:todd@cloudera.com<ma...@cloudera.com>]
Sent: Monday, April 12, 2010 8:18 PM
To: hdfs-user@hadoop.apache.org<ma...@hadoop.apache.org>; gokulm@huawei.com<ma...@huawei.com>
Subject: Re: response time increases lineraly for each thread!
Can you be more specific about your benchmark? You're almost certainly being limited by either outgoing network from your single node, or by disk throughput on that node (writes will go local in addition to remote replicas if you are running a DN on the same host)
-Todd
On Mon, Apr 12, 2010 at 7:21 AM, Gokulakannan M <go...@huawei.com>> wrote:
Hi,
When I tested the performance of a 11 node hadoop cluster(private nw) for the write scenario with several threads, I noticed that the time for each write increases linearly(what ever the file size may be).
Actually I'm running these threads as a single process in the system where namenode is running. I'm getting inconsistent results when I run this suite each time. The results are fluctuating. What might be the bottlenecks?
I think running this in parallel by spawning multiple shells will give consistent result. But running these in the same system will give exact results??
Any ways to measure the performance correctly?
Appreciated any help in this.
Thanks,
Gokul
--
Todd Lipcon
Software Engineer, Cloudera
--
Todd Lipcon
Software Engineer, Cloudera
--
Todd Lipcon
Software Engineer, Cloudera
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
RE: response time increases lineraly for each thread!
Posted by Gokulakannan M <go...@huawei.com>.
Hi Todd,
This really makes sense. Thanks :-)
Thanks,
Gokul
_____
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Wednesday, April 14, 2010 2:04 AM
To: gokulm@huawei.com
Cc: hdfs-user@hadoop.apache.org; sagar_shukla@persistent.co.in
Subject: Re: response time increases lineraly for each thread!
Hi Gokul,
This linear degradation makes sense. If you're saturating your outgoing
network bandwidth from the node that's doing the writing, then doubling the
number of writers will decrease the amount of bandwidth available to each
writer by a factor of two. So of course each writer will take twice as long
if there are twice as many writers.
This is true regardless of whether you are disk bound or network bound on
the local node - you have a finite resource (throughput) which you're
splitting among more clients.
If you run the concurrent writers from different nodes you'll probably see
that the write time stays constant until you start to saturate bandwidth on
all of the nodes.
Does this make sense?
-Todd
On Tue, Apr 13, 2010 at 2:15 AM, Gokulakannan M <go...@huawei.com> wrote:
Todd,
I am benchmarking the performance for the write scenario(Not
only for single user but for concurrent users). I've implemented a simple
FileWrite() method that takes namenode address and source file and write
into hdfs. I'm measuring the response time by just getting the time diff
before and after calling the FileWrite() method (by using
System.currentMillliSeconds()).I've created several threads for this
FileWrite() method and measure the performance. Is anything wrong with the
test code?
No I didn't configure dfs.data.dir to all volumes but the
current one. So the issue might be due to disk latency? The total space
available is 750GB(seen from name UI).
Thanks,
Gokul
_____
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Tuesday, April 13, 2010 10:54 AM
To: gokulm@huawei.com; hdfs-user@hadoop.apache.org
Cc: sagar_shukla@persistent.co.in
Subject: Re: response time increases lineraly for each thread!
Hi Gokul,
Would it be possible to post your benchmark code somewhere? I'm not clear on
what exact thing you're measuring - IOUtils.copyBytes is a pretty general
method.
Also, what disk hardware are you using? Have you configured dfs.data.dir to
point to all local drives on each DN? Could very well be a factor of disk
seeks limiting you.
-Todd
On Mon, Apr 12, 2010 at 10:18 PM, Gokulakannan M <go...@huawei.com> wrote:
@Todd,
I am using 1 namenode and 10 datanodes. Replication factor is 3.
These are the details about my test suite. It uses the simple
write code that uses IOUtils.copyBytes() and calculates the response time
for each write for a given number of users(threads).
From namenode, I am running the test suite which calculates the
response time for writing a file for several concurrent users(1 , 4 , 9 ,
20 ). Whatever the file size may be(I tried with 100 MB , 1 GB and 5 GB
files), I find that the response time increases linearly.
These are the results for a 100 MB file.
No of Threads
Response time(milli sec)
1
2199
4
2763
2774
34766
64488
9
3620
4897
5018
9991
65501
124156
183639
243631
314233
20
6702
6784
6985
8987
9202
9271
9925
10752
68237
70878
73469
75261
77507
82061
129942
137098
141822
194199
269353
322328
You can see the write time increases for each thread.
@ Sagar:
I think network is not the issue here. This is a private network
with high bandwidth. Moreover I tried the running the same test suite with
several configurations(pseudo dist mode, 1 NN 1 DN Cluster , 1 NN 3 DN
cluster) in my usual network(typically with low bandwidth) and obtained the
results like above :(
Thanks,
Gokul
_____
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Monday, April 12, 2010 8:18 PM
To: hdfs-user@hadoop.apache.org; gokulm@huawei.com
Subject: Re: response time increases lineraly for each thread!
Can you be more specific about your benchmark? You're almost certainly being
limited by either outgoing network from your single node, or by disk
throughput on that node (writes will go local in addition to remote replicas
if you are running a DN on the same host)
-Todd
On Mon, Apr 12, 2010 at 7:21 AM, Gokulakannan M <go...@huawei.com> wrote:
Hi,
When I tested the performance of a 11 node hadoop
cluster(private nw) for the write scenario with several threads, I noticed
that the time for each write increases linearly(what ever the file size may
be).
Actually I'm running these threads as a single process in the
system where namenode is running. I'm getting inconsistent results when I
run this suite each time. The results are fluctuating. What might be the
bottlenecks?
I think running this in parallel by spawning multiple shells
will give consistent result. But running these in the same system will give
exact results??
Any ways to measure the performance correctly?
Appreciated any help in this.
Thanks,
Gokul
--
Todd Lipcon
Software Engineer, Cloudera
--
Todd Lipcon
Software Engineer, Cloudera
--
Todd Lipcon
Software Engineer, Cloudera
Re: response time increases lineraly for each thread!
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Gokul,
This linear degradation makes sense. If you're saturating your outgoing
network bandwidth from the node that's doing the writing, then doubling the
number of writers will decrease the amount of bandwidth available to each
writer by a factor of two. So of course each writer will take twice as long
if there are twice as many writers.
This is true regardless of whether you are disk bound or network bound on
the local node - you have a finite resource (throughput) which you're
splitting among more clients.
If you run the concurrent writers from different nodes you'll probably see
that the write time stays constant until you start to saturate bandwidth on
all of the nodes.
Does this make sense?
-Todd
On Tue, Apr 13, 2010 at 2:15 AM, Gokulakannan M <go...@huawei.com> wrote:
> Todd,
>
>
>
> I am benchmarking the performance for the write scenario(Not
> only for single user but for concurrent users). I've implemented a simple
> FileWrite() method that takes namenode address and source file and write
> into hdfs. I'm measuring the response time by just getting the time diff
> before and after calling the FileWrite() method (by using
> System.currentMillliSeconds()).I've created several threads for this
> FileWrite() method and measure the performance. Is anything wrong with the
> test code?
>
>
>
>
>
> No I didn't configure dfs.data.dir to all volumes but the
> current one. So the issue might be due to disk latency? The total space
> available is 750GB(seen from name UI).
>
>
>
> Thanks,
>
> Gokul
>
>
>
>
> ------------------------------
>
> *From:* Todd Lipcon [mailto:todd@cloudera.com]
> *Sent:* Tuesday, April 13, 2010 10:54 AM
> *To:* gokulm@huawei.com; hdfs-user@hadoop.apache.org
> *Cc:* sagar_shukla@persistent.co.in
>
> *Subject:* Re: response time increases lineraly for each thread!
>
>
>
> Hi Gokul,
>
>
>
> Would it be possible to post your benchmark code somewhere? I'm not clear
> on what exact thing you're measuring - IOUtils.copyBytes is a pretty general
> method.
>
>
>
> Also, what disk hardware are you using? Have you configured dfs.data.dir to
> point to all local drives on each DN? Could very well be a factor of disk
> seeks limiting you.
>
> -Todd
>
>
>
>
>
> On Mon, Apr 12, 2010 at 10:18 PM, Gokulakannan M <go...@huawei.com>
> wrote:
>
> @Todd,
>
>
>
> I am using 1 namenode and 10 datanodes. Replication factor is
> 3.
>
>
>
> These are the details about my test suite. It uses the simple
> write code that uses IOUtils.copyBytes() and calculates the response time
> for each write for a given number of users(threads).
>
>
>
> From namenode, I am running the test suite which calculates the
> response time for writing a file for several concurrent users(1 , 4 , 9 ,
> 20 ). Whatever the file size may be(I tried with 100 MB , 1 GB and 5 GB
> files), I find that the response time increases linearly.
>
>
>
> These are the results for a 100 MB file.
>
>
>
>
>
> No of Threads
>
> Response time(milli sec)
>
> 1
>
> 2199
>
>
>
>
>
> 4
>
> 2763
>
>
>
> 2774
>
>
>
> 34766
>
>
>
> 64488
>
>
>
>
>
> 9
>
> 3620
>
>
>
> 4897
>
>
>
> 5018
>
>
>
> 9991
>
>
>
> 65501
>
>
>
> 124156
>
>
>
> 183639
>
>
>
> 243631
>
>
>
> 314233
>
>
>
>
>
> 20
>
> 6702
>
>
>
> 6784
>
>
>
> 6985
>
>
>
> 8987
>
>
>
> 9202
>
>
>
> 9271
>
>
>
> 9925
>
>
>
> 10752
>
>
>
> 68237
>
>
>
> 70878
>
>
>
> 73469
>
>
>
> 75261
>
>
>
> 77507
>
>
>
> 82061
>
>
>
> 129942
>
>
>
> 137098
>
>
>
> 141822
>
>
>
> 194199
>
>
>
> 269353
>
>
>
> 322328
>
>
>
>
>
> You can see the write time increases for each thread.
>
>
>
> @ Sagar:
>
>
>
> I think network is not the issue here. This is a private
> network with high bandwidth. Moreover I tried the running the same test
> suite with several configurations(pseudo dist mode, 1 NN 1 DN Cluster , 1 NN
> 3 DN cluster) in my usual network(typically with low bandwidth) and obtained
> the results like above :(
>
>
>
>
>
> Thanks,
>
> Gokul
>
>
>
>
>
>
> ------------------------------
>
> *From:* Todd Lipcon [mailto:todd@cloudera.com]
> *Sent:* Monday, April 12, 2010 8:18 PM
> *To:* hdfs-user@hadoop.apache.org; gokulm@huawei.com
> *Subject:* Re: response time increases lineraly for each thread!
>
>
>
> Can you be more specific about your benchmark? You're almost certainly
> being limited by either outgoing network from your single node, or by disk
> throughput on that node (writes will go local in addition to remote replicas
> if you are running a DN on the same host)
>
>
>
> -Todd
>
> On Mon, Apr 12, 2010 at 7:21 AM, Gokulakannan M <go...@huawei.com> wrote:
>
>
>
> Hi,
>
>
>
> When I tested the performance of a 11 node hadoop
> cluster(private nw) for the write scenario with several threads, I noticed
> that the time for each write increases linearly(what ever the file size may
> be).
>
>
>
> Actually I'm running these threads as a single process in the
> system where namenode is running. I'm getting inconsistent results when I
> run this suite each time. The results are fluctuating. What might be the
> bottlenecks?
>
>
>
> I think running this in parallel by spawning multiple shells
> will give consistent result. But running these in the same system will give
> exact results??
>
>
>
> *Any* *ways to measure the performance correctly?*
>
> * *
>
> * *Appreciated any help in this.
>
>
>
> Thanks,
>
> Gokul
>
>
>
>
>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Todd Lipcon
Software Engineer, Cloudera
RE: response time increases lineraly for each thread!
Posted by Gokulakannan M <go...@huawei.com>.
Todd,
I am benchmarking the performance for the write scenario(Not
only for single user but for concurrent users). I've implemented a simple
FileWrite() method that takes namenode address and source file and write
into hdfs. I'm measuring the response time by just getting the time diff
before and after calling the FileWrite() method (by using
System.currentMillliSeconds()).I've created several threads for this
FileWrite() method and measure the performance. Is anything wrong with the
test code?
No I didn't configure dfs.data.dir to all volumes but the
current one. So the issue might be due to disk latency? The total space
available is 750GB(seen from name UI).
Thanks,
Gokul
_____
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Tuesday, April 13, 2010 10:54 AM
To: gokulm@huawei.com; hdfs-user@hadoop.apache.org
Cc: sagar_shukla@persistent.co.in
Subject: Re: response time increases lineraly for each thread!
Hi Gokul,
Would it be possible to post your benchmark code somewhere? I'm not clear on
what exact thing you're measuring - IOUtils.copyBytes is a pretty general
method.
Also, what disk hardware are you using? Have you configured dfs.data.dir to
point to all local drives on each DN? Could very well be a factor of disk
seeks limiting you.
-Todd
On Mon, Apr 12, 2010 at 10:18 PM, Gokulakannan M <go...@huawei.com> wrote:
@Todd,
I am using 1 namenode and 10 datanodes. Replication factor is 3.
These are the details about my test suite. It uses the simple
write code that uses IOUtils.copyBytes() and calculates the response time
for each write for a given number of users(threads).
From namenode, I am running the test suite which calculates the
response time for writing a file for several concurrent users(1 , 4 , 9 ,
20 ). Whatever the file size may be(I tried with 100 MB , 1 GB and 5 GB
files), I find that the response time increases linearly.
These are the results for a 100 MB file.
No of Threads
Response time(milli sec)
1
2199
4
2763
2774
34766
64488
9
3620
4897
5018
9991
65501
124156
183639
243631
314233
20
6702
6784
6985
8987
9202
9271
9925
10752
68237
70878
73469
75261
77507
82061
129942
137098
141822
194199
269353
322328
You can see the write time increases for each thread.
@ Sagar:
I think network is not the issue here. This is a private network
with high bandwidth. Moreover I tried the running the same test suite with
several configurations(pseudo dist mode, 1 NN 1 DN Cluster , 1 NN 3 DN
cluster) in my usual network(typically with low bandwidth) and obtained the
results like above :(
Thanks,
Gokul
_____
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Monday, April 12, 2010 8:18 PM
To: hdfs-user@hadoop.apache.org; gokulm@huawei.com
Subject: Re: response time increases lineraly for each thread!
Can you be more specific about your benchmark? You're almost certainly being
limited by either outgoing network from your single node, or by disk
throughput on that node (writes will go local in addition to remote replicas
if you are running a DN on the same host)
-Todd
On Mon, Apr 12, 2010 at 7:21 AM, Gokulakannan M <go...@huawei.com> wrote:
Hi,
When I tested the performance of a 11 node hadoop
cluster(private nw) for the write scenario with several threads, I noticed
that the time for each write increases linearly(what ever the file size may
be).
Actually I'm running these threads as a single process in the
system where namenode is running. I'm getting inconsistent results when I
run this suite each time. The results are fluctuating. What might be the
bottlenecks?
I think running this in parallel by spawning multiple shells
will give consistent result. But running these in the same system will give
exact results??
Any ways to measure the performance correctly?
Appreciated any help in this.
Thanks,
Gokul
--
Todd Lipcon
Software Engineer, Cloudera
--
Todd Lipcon
Software Engineer, Cloudera
Re: response time increases lineraly for each thread!
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Gokul,
Would it be possible to post your benchmark code somewhere? I'm not clear on
what exact thing you're measuring - IOUtils.copyBytes is a pretty general
method.
Also, what disk hardware are you using? Have you configured dfs.data.dir to
point to all local drives on each DN? Could very well be a factor of disk
seeks limiting you.
-Todd
On Mon, Apr 12, 2010 at 10:18 PM, Gokulakannan M <go...@huawei.com> wrote:
> @Todd,
>
>
>
> I am using 1 namenode and 10 datanodes. Replication factor is
> 3.
>
>
>
> These are the details about my test suite. It uses the simple
> write code that uses IOUtils.copyBytes() and calculates the response time
> for each write for a given number of users(threads).
>
>
>
> From namenode, I am running the test suite which calculates the
> response time for writing a file for several concurrent users(1 , 4 , 9 ,
> 20 ). Whatever the file size may be(I tried with 100 MB , 1 GB and 5 GB
> files), I find that the response time increases linearly.
>
>
>
> These are the results for a 100 MB file.
>
>
>
>
>
> No of Threads
>
> Response time(milli sec)
>
> 1
>
> 2199
>
>
>
>
>
> 4
>
> 2763
>
>
>
> 2774
>
>
>
> 34766
>
>
>
> 64488
>
>
>
>
>
> 9
>
> 3620
>
>
>
> 4897
>
>
>
> 5018
>
>
>
> 9991
>
>
>
> 65501
>
>
>
> 124156
>
>
>
> 183639
>
>
>
> 243631
>
>
>
> 314233
>
>
>
>
>
> 20
>
> 6702
>
>
>
> 6784
>
>
>
> 6985
>
>
>
> 8987
>
>
>
> 9202
>
>
>
> 9271
>
>
>
> 9925
>
>
>
> 10752
>
>
>
> 68237
>
>
>
> 70878
>
>
>
> 73469
>
>
>
> 75261
>
>
>
> 77507
>
>
>
> 82061
>
>
>
> 129942
>
>
>
> 137098
>
>
>
> 141822
>
>
>
> 194199
>
>
>
> 269353
>
>
>
> 322328
>
>
>
>
>
> You can see the write time increases for each thread.
>
>
>
> @ Sagar:
>
>
>
> I think network is not the issue here. This is a private
> network with high bandwidth. Moreover I tried the running the same test
> suite with several configurations(pseudo dist mode, 1 NN 1 DN Cluster , 1 NN
> 3 DN cluster) in my usual network(typically with low bandwidth) and obtained
> the results like above :(
>
>
>
>
>
> Thanks,
>
> Gokul
>
>
>
>
>
>
> ------------------------------
>
> *From:* Todd Lipcon [mailto:todd@cloudera.com]
> *Sent:* Monday, April 12, 2010 8:18 PM
> *To:* hdfs-user@hadoop.apache.org; gokulm@huawei.com
> *Subject:* Re: response time increases lineraly for each thread!
>
>
>
> Can you be more specific about your benchmark? You're almost certainly
> being limited by either outgoing network from your single node, or by disk
> throughput on that node (writes will go local in addition to remote replicas
> if you are running a DN on the same host)
>
>
>
> -Todd
>
> On Mon, Apr 12, 2010 at 7:21 AM, Gokulakannan M <go...@huawei.com> wrote:
>
>
>
> Hi,
>
>
>
> When I tested the performance of a 11 node hadoop
> cluster(private nw) for the write scenario with several threads, I noticed
> that the time for each write increases linearly(what ever the file size may
> be).
>
>
>
> Actually I'm running these threads as a single process in the
> system where namenode is running. I'm getting inconsistent results when I
> run this suite each time. The results are fluctuating. What might be the
> bottlenecks?
>
>
>
> I think running this in parallel by spawning multiple shells
> will give consistent result. But running these in the same system will give
> exact results??
>
>
>
> *Any* *ways to measure the performance correctly?*
>
> * *
>
> * *Appreciated any help in this.
>
>
>
> Thanks,
>
> Gokul
>
>
>
>
>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Todd Lipcon
Software Engineer, Cloudera
RE: response time increases lineraly for each thread!
Posted by Gokulakannan M <go...@huawei.com>.
@Todd,
I am using 1 namenode and 10 datanodes. Replication factor is 3.
These are the details about my test suite. It uses the simple
write code that uses IOUtils.copyBytes() and calculates the response time
for each write for a given number of users(threads).
From namenode, I am running the test suite which calculates the
response time for writing a file for several concurrent users(1 , 4 , 9 ,
20 ). Whatever the file size may be(I tried with 100 MB , 1 GB and 5 GB
files), I find that the response time increases linearly.
These are the results for a 100 MB file.
No of Threads
Response time(milli sec)
1
2199
4
2763
2774
34766
64488
9
3620
4897
5018
9991
65501
124156
183639
243631
314233
20
6702
6784
6985
8987
9202
9271
9925
10752
68237
70878
73469
75261
77507
82061
129942
137098
141822
194199
269353
322328
You can see the write time increases for each thread.
@ Sagar:
I think network is not the issue here. This is a private network
with high bandwidth. Moreover I tried the running the same test suite with
several configurations(pseudo dist mode, 1 NN 1 DN Cluster , 1 NN 3 DN
cluster) in my usual network(typically with low bandwidth) and obtained the
results like above :(
Thanks,
Gokul
_____
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Monday, April 12, 2010 8:18 PM
To: hdfs-user@hadoop.apache.org; gokulm@huawei.com
Subject: Re: response time increases lineraly for each thread!
Can you be more specific about your benchmark? You're almost certainly being
limited by either outgoing network from your single node, or by disk
throughput on that node (writes will go local in addition to remote replicas
if you are running a DN on the same host)
-Todd
On Mon, Apr 12, 2010 at 7:21 AM, Gokulakannan M <go...@huawei.com> wrote:
Hi,
When I tested the performance of a 11 node hadoop
cluster(private nw) for the write scenario with several threads, I noticed
that the time for each write increases linearly(what ever the file size may
be).
Actually I'm running these threads as a single process in the
system where namenode is running. I'm getting inconsistent results when I
run this suite each time. The results are fluctuating. What might be the
bottlenecks?
I think running this in parallel by spawning multiple shells
will give consistent result. But running these in the same system will give
exact results??
Any ways to measure the performance correctly?
Appreciated any help in this.
Thanks,
Gokul
--
Todd Lipcon
Software Engineer, Cloudera
Re: response time increases lineraly for each thread!
Posted by Todd Lipcon <to...@cloudera.com>.
Can you be more specific about your benchmark? You're almost certainly being
limited by either outgoing network from your single node, or by disk
throughput on that node (writes will go local in addition to remote replicas
if you are running a DN on the same host)
-Todd
On Mon, Apr 12, 2010 at 7:21 AM, Gokulakannan M <go...@huawei.com> wrote:
>
>
> Hi,
>
>
>
> When I tested the performance of a 11 node hadoop
> cluster(private nw) for the write scenario with several threads, I noticed
> that the time for each write increases linearly(what ever the file size may
> be).
>
>
>
> Actually I'm running these threads as a single process in the
> system where namenode is running. I'm getting inconsistent results when I
> run this suite each time. The results are fluctuating. What might be the
> bottlenecks?
>
>
>
> I think running this in parallel by spawning multiple shells
> will give consistent result. But running these in the same system will give
> exact results??
>
>
>
> *Any* *ways to measure the performance correctly?*
>
> * *
>
> * *Appreciated any help in this.
>
>
>
> Thanks,
>
> Gokul
>
>
>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
RE: response time increases lineraly for each thread!
Posted by Sagar Shukla <sa...@persistent.co.in>.
Hi Gokul,
What is the network topology ? Is it a 1Gbps or 10Gbps n/w ? It is possible that network could be getting choked up causing the write time to increase.
Thanks,
Sagar
From: Gokulakannan M [mailto:gokulm@huawei.com]
Sent: Monday, April 12, 2010 7:52 PM
To: hdfs-user@hadoop.apache.org
Subject: response time increases lineraly for each thread!
Hi,
When I tested the performance of a 11 node hadoop cluster(private nw) for the write scenario with several threads, I noticed that the time for each write increases linearly(what ever the file size may be).
Actually I'm running these threads as a single process in the system where namenode is running. I'm getting inconsistent results when I run this suite each time. The results are fluctuating. What might be the bottlenecks?
I think running this in parallel by spawning multiple shells will give consistent result. But running these in the same system will give exact results??
Any ways to measure the performance correctly?
Appreciated any help in this.
Thanks,
Gokul
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.