You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by stephen mulcahy <st...@deri.org> on 2009/05/26 12:10:05 UTC
Bisection bandwidth needed for a small Hadoop cluster
Hi,
Has anyone here investigated what level of bisection bandwidth is needed
for a Hadoop cluster which spans more than one rack?
I'm currently sizing and planning a new Hadoop cluster and I'm wondering
what the performance implications will be if we end up with a cluster
spread across two racks. I'd expect we'll have one 48-port gigabit
switch in each 42u rack. If we end up with 60 systems spread across
these two switches - how much bandwidth should I have between the racks?
I'll have 6 gigabit ports available for links between racks - i.e. up to
6 Gbps. Would this be sufficient bisection bandwidth for Hadoop or
should I be considering increased bandwidth between racks (maybe using
fibre links between the switches or introducing another switch)?
Thanks for any thoughts on this.
-stephen
--
Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
http://di2.deri.ie http://webstar.deri.ie http://sindice.com
Re: Bisection bandwidth needed for a small Hadoop cluster
Posted by stephen mulcahy <st...@deri.org>.
Todd Lipcon wrote:
> Hi Stephen,
>
> The true answer depends on the types of jobs you're running. As a back of
> the envelope calculation I might figure something like this:
> ...
> So, the simple answer is that 4-6Gbps is most likely just fine for most
> practical jobs. If you want to be extra safe, many inexpensive switches can
> operate in a "stacked" configuration where the bandwidth between them is
> essentially backplane speed. That should scale you to 96 nodes with plenty
> of headroom.
Hi Todd,
Many thanks for the detailed response - I understand it all depends on
my specific job and that the usual YMMV caveat applies but your figures
are very helpful to give me some ballpark figures to play with.
From what I can see, the inexpensive switches use add-on modules in
stacking configurations and those add-on modules are reasonably pricey
(especially if you want to stack switches with 10 Gbps links rather than
1 Gbps links) so the ballpark number is pretty useful from a budgeting
perspective.
Thanks,
-stephen
--
Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
http://di2.deri.ie http://webstar.deri.ie http://sindice.com
Re: Bisection bandwidth needed for a small Hadoop cluster
Posted by Amr Awadallah <aa...@cloudera.com>.
Stephen,
I highly recommend you get switches that have a few 10GigE ports on
them for interlinking.
So say you have 40 servers/rack, then get top-of-rack switches with
forty 1GigE ports and two 10GigE ports, this way you can have half the
servers talking to the other rack without bottlenecks. The prices for
such switches have been dropping significantly in recent months (you
should be able to get them for sub $10K). Take a look at the switches
from Arista Networks (www.aristanetworks.com) for competitive pricing.
Cheers,
-- amr
Todd Lipcon wrote:
> Hi Stephen,
>
> The true answer depends on the types of jobs you're running. As a back of
> the envelope calculation I might figure something like this:
>
> 60 nodes total = 30 nodes per rack
> Each node might process about 100MB/sec of data
> In the case of a sort job where the intermediate data is the same size as
> the input data, that means each node needs to shuffle 100MB/sec of data
> In aggregate, each rack is then producing about 3GB/sec of data
> However, given even reducer spread across the racks, each rack will need to
> send 1.5GB/sec to reducers running on the other rack.
> Since the connection is full duplex, that means you need 1.5GB/sec of
> bisection bandwidth for this theoretical job. So that's 12Gbps.
>
> However, the above calculations are probably somewhat of an upper bound. A
> large number of jobs have significant data reduction during the map phase,
> either by some kind of filtering/selection going on in the Mapper itself, or
> by good usage of Combiners. Additionally, intermediate data compression can
> cut the intermediate data transfer by a significant factor. Lastly, although
> your disks can probably provide 100MB sustained throughput, it's rare to see
> a MR job which can sustain disk speed IO through the entire pipeline. So,
> I'd say my estimate is at least a factor of 2 too high.
>
> So, the simple answer is that 4-6Gbps is most likely just fine for most
> practical jobs. If you want to be extra safe, many inexpensive switches can
> operate in a "stacked" configuration where the bandwidth between them is
> essentially backplane speed. That should scale you to 96 nodes with plenty
> of headroom.
>
> -Todd
>
> On Tue, May 26, 2009 at 3:10 AM, stephen mulcahy
> <st...@deri.org>wrote:
>
>
>> Hi,
>>
>> Has anyone here investigated what level of bisection bandwidth is needed
>> for a Hadoop cluster which spans more than one rack?
>>
>> I'm currently sizing and planning a new Hadoop cluster and I'm wondering
>> what the performance implications will be if we end up with a cluster spread
>> across two racks. I'd expect we'll have one 48-port gigabit switch in each
>> 42u rack. If we end up with 60 systems spread across these two switches -
>> how much bandwidth should I have between the racks?
>>
>> I'll have 6 gigabit ports available for links between racks - i.e. up to 6
>> Gbps. Would this be sufficient bisection bandwidth for Hadoop or should I be
>> considering increased bandwidth between racks (maybe using fibre links
>> between the switches or introducing another switch)?
>>
>> Thanks for any thoughts on this.
>>
>> -stephen
>>
>> --
>> Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
>> NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
>> http://di2.deri.ie http://webstar.deri.ie http://sindice.com
>>
>>
>
>
Re: Bisection bandwidth needed for a small Hadoop cluster
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Stephen,
The true answer depends on the types of jobs you're running. As a back of
the envelope calculation I might figure something like this:
60 nodes total = 30 nodes per rack
Each node might process about 100MB/sec of data
In the case of a sort job where the intermediate data is the same size as
the input data, that means each node needs to shuffle 100MB/sec of data
In aggregate, each rack is then producing about 3GB/sec of data
However, given even reducer spread across the racks, each rack will need to
send 1.5GB/sec to reducers running on the other rack.
Since the connection is full duplex, that means you need 1.5GB/sec of
bisection bandwidth for this theoretical job. So that's 12Gbps.
However, the above calculations are probably somewhat of an upper bound. A
large number of jobs have significant data reduction during the map phase,
either by some kind of filtering/selection going on in the Mapper itself, or
by good usage of Combiners. Additionally, intermediate data compression can
cut the intermediate data transfer by a significant factor. Lastly, although
your disks can probably provide 100MB sustained throughput, it's rare to see
a MR job which can sustain disk speed IO through the entire pipeline. So,
I'd say my estimate is at least a factor of 2 too high.
So, the simple answer is that 4-6Gbps is most likely just fine for most
practical jobs. If you want to be extra safe, many inexpensive switches can
operate in a "stacked" configuration where the bandwidth between them is
essentially backplane speed. That should scale you to 96 nodes with plenty
of headroom.
-Todd
On Tue, May 26, 2009 at 3:10 AM, stephen mulcahy
<st...@deri.org>wrote:
> Hi,
>
> Has anyone here investigated what level of bisection bandwidth is needed
> for a Hadoop cluster which spans more than one rack?
>
> I'm currently sizing and planning a new Hadoop cluster and I'm wondering
> what the performance implications will be if we end up with a cluster spread
> across two racks. I'd expect we'll have one 48-port gigabit switch in each
> 42u rack. If we end up with 60 systems spread across these two switches -
> how much bandwidth should I have between the racks?
>
> I'll have 6 gigabit ports available for links between racks - i.e. up to 6
> Gbps. Would this be sufficient bisection bandwidth for Hadoop or should I be
> considering increased bandwidth between racks (maybe using fibre links
> between the switches or introducing another switch)?
>
> Thanks for any thoughts on this.
>
> -stephen
>
> --
> Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
> NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
> http://di2.deri.ie http://webstar.deri.ie http://sindice.com
>