You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jonathan Disher <jd...@parad.net> on 2011/05/10 14:50:03 UTC
HDFS network bottleneck - namenode?
I will preface this with a couple statements: a) it's almost 6am, and I've been up all night b) I'm drugged up from an allergic reaction, so I may not be firing on all 64 bits.
Do I correctly understand the HDFS architecture in that the namenode is a network bottleneck into the system? I.e., it doesn't really matter how many ethernet interfaces I roll into my data nodes, I will always be limited in how much traffic I can drive to the HDFS pool by the network capacity of the namenode?
I am trying to move a -lot- of data, and i'd like to not throttle the namenode (especially in the old cluster, where I cannot just bond up more interfaces). If there's a way to spread the inbound network (for block writes) traffic I'd love to hear it.
Thanks!
-j
Re: HDFS network bottleneck - namenode?
Posted by Jonathan Disher <jd...@parad.net>.
Okay, thanks. I -hoped- it was this way.
Sadly, all my files are small (the largest are around 40MB). But oh well!
-j
On May 10, 2011, at 10:46 AM, Matthew Foley wrote:
> Will's right, meta-data transactions go through the Namenode, but all the content data
> read/write activity is directly between Clients and Datanodes, and replication activity is
> Datanode-to-Datanode. No bottlenecks, as long as your Namenode has enough ram to
> hold the namespace in memory, and enough cores to handle a modestly high transaction
> rate.
>
> And if the individual data files are large (Hadoop-scale "large", that is :-) ), you can even
> decrease the meta-data/data ratio by increasing the block size from the default 32MB
> to 64MB or even 128MB.
>
> --Matt
>
>
> On May 10, 2011, at 6:03 AM, Will Maier wrote:
>
> Hi Jonathan-
>
> On Tue, May 10, 2011 at 05:50:03AM -0700, Jonathan Disher wrote:
>> I will preface this with a couple statements: a) it's almost 6am, and I've
>> been up all night b) I'm drugged up from an allergic reaction, so I may not be
>> firing on all 64 bits.
>>
>> Do I correctly understand the HDFS architecture in that the namenode is a
>> network bottleneck into the system? I.e., it doesn't really matter how many
>> ethernet interfaces I roll into my data nodes, I will always be limited in
>> how much traffic I can drive to the HDFS pool by the network capacity of the
>> namenode?
>
> No. This diagram should help:
>
> http://hadoop.apache.org/hdfs/docs/current/hdfs_design.html#NameNode+and+DataNodes
>
> The Namenode is a single point of failure, not (under most imaginable
> conditions) a bottleneck.
>
>> I am trying to move a -lot- of data, and i'd like to not throttle the namenode
>> (especially in the old cluster, where I cannot just bond up more interfaces).
>> If there's a way to spread the inbound network (for block writes) traffic I'd
>> love to hear it.
>
> During our (highly distributed) migration, we were writing into HDFS at up to 5 GB/s.
> The more datanodes and writers you have, the faster your aggregate throughput.
>
> --
>
> Will Maier - UW High Energy Physics
> cel: 608.438.6162
> tel: 608.263.9692
> web: http://www.hep.wisc.edu/~wcmaier/
Re: HDFS network bottleneck - namenode?
Posted by Matthew Foley <ma...@yahoo-inc.com>.
Will's right, meta-data transactions go through the Namenode, but all the content data
read/write activity is directly between Clients and Datanodes, and replication activity is
Datanode-to-Datanode. No bottlenecks, as long as your Namenode has enough ram to
hold the namespace in memory, and enough cores to handle a modestly high transaction
rate.
And if the individual data files are large (Hadoop-scale "large", that is :-) ), you can even
decrease the meta-data/data ratio by increasing the block size from the default 32MB
to 64MB or even 128MB.
--Matt
On May 10, 2011, at 6:03 AM, Will Maier wrote:
Hi Jonathan-
On Tue, May 10, 2011 at 05:50:03AM -0700, Jonathan Disher wrote:
> I will preface this with a couple statements: a) it's almost 6am, and I've
> been up all night b) I'm drugged up from an allergic reaction, so I may not be
> firing on all 64 bits.
>
> Do I correctly understand the HDFS architecture in that the namenode is a
> network bottleneck into the system? I.e., it doesn't really matter how many
> ethernet interfaces I roll into my data nodes, I will always be limited in
> how much traffic I can drive to the HDFS pool by the network capacity of the
> namenode?
No. This diagram should help:
http://hadoop.apache.org/hdfs/docs/current/hdfs_design.html#NameNode+and+DataNodes
The Namenode is a single point of failure, not (under most imaginable
conditions) a bottleneck.
> I am trying to move a -lot- of data, and i'd like to not throttle the namenode
> (especially in the old cluster, where I cannot just bond up more interfaces).
> If there's a way to spread the inbound network (for block writes) traffic I'd
> love to hear it.
During our (highly distributed) migration, we were writing into HDFS at up to 5 GB/s.
The more datanodes and writers you have, the faster your aggregate throughput.
--
Will Maier - UW High Energy Physics
cel: 608.438.6162
tel: 608.263.9692
web: http://www.hep.wisc.edu/~wcmaier/
Re: HDFS network bottleneck - namenode?
Posted by Will Maier <wc...@hep.wisc.edu>.
Hi Jonathan-
On Tue, May 10, 2011 at 05:50:03AM -0700, Jonathan Disher wrote:
> I will preface this with a couple statements: a) it's almost 6am, and I've
> been up all night b) I'm drugged up from an allergic reaction, so I may not be
> firing on all 64 bits.
>
> Do I correctly understand the HDFS architecture in that the namenode is a
> network bottleneck into the system? I.e., it doesn't really matter how many
> ethernet interfaces I roll into my data nodes, I will always be limited in
> how much traffic I can drive to the HDFS pool by the network capacity of the
> namenode?
No. This diagram should help:
http://hadoop.apache.org/hdfs/docs/current/hdfs_design.html#NameNode+and+DataNodes
The Namenode is a single point of failure, not (under most imaginable
conditions) a bottleneck.
> I am trying to move a -lot- of data, and i'd like to not throttle the namenode
> (especially in the old cluster, where I cannot just bond up more interfaces).
> If there's a way to spread the inbound network (for block writes) traffic I'd
> love to hear it.
During our (highly distributed) migration, we were writing into HDFS at up to 5 GB/s.
The more datanodes and writers you have, the faster your aggregate throughput.
--
Will Maier - UW High Energy Physics
cel: 608.438.6162
tel: 608.263.9692
web: http://www.hep.wisc.edu/~wcmaier/