You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Buttler, David" <bu...@llnl.gov> on 2010/08/20 17:36:25 UTC
Taking advantage of multiple networks
Hi all,
I have a small hbase cluster that I have recently filled with about 500M records (some of them quite large). One of the things that I notice when I do different types of map / reduce jobs over my table is that the network becomes a bottleneck. Currently I am running single gig Ethernet on this cluster, but it has 4 network ports.
My question is this: is it possible to set up hadoop/hbase to take advantage of multiple networks connecting the computers?
Could I specify multiple network connections in the config file?
Would it make sense to put the region servers on a different network than the data nodes?
Would it be more efficient to bond multiple channels at the OS level?
Thanks for the suggestions,
Dave
RE: Taking advantage of multiple networks
Posted by Michael Segel <mi...@hotmail.com>.
> From: alexey@kovyrin.net
> Date: Fri, 20 Aug 2010 11:48:51 -0400
> Subject: Re: Taking advantage of multiple networks
> To: user@hbase.apache.org
>
> Not sure about hadoop support, but OS-level (or maybe even network
> card level) port bonding looks like the way to go.
>
Yes, port bonding is possible and should be 'invisible' to HBase.
Do you have Ganglia installed and if so, what do you see with respect to network traffic when your large jobs are running?
It could be that the network isn't the bottleneck but a contributing factor.
-Mike
> On Fri, Aug 20, 2010 at 11:36 AM, Buttler, David <bu...@llnl.gov> wrote:
> > Hi all,
> > I have a small hbase cluster that I have recently filled with about 500M records (some of them quite large). One of the things that I notice when I do different types of map / reduce jobs over my table is that the network becomes a bottleneck. Currently I am running single gig Ethernet on this cluster, but it has 4 network ports.
> >
> > My question is this: is it possible to set up hadoop/hbase to take advantage of multiple networks connecting the computers?
> > Could I specify multiple network connections in the config file?
> > Would it make sense to put the region servers on a different network than the data nodes?
> > Would it be more efficient to bond multiple channels at the OS level?
>
> --
> Alexey Kovyrin
> http://kovyrin.net/
Re: Taking advantage of multiple networks
Posted by Alexey Kovyrin <al...@kovyrin.net>.
Not sure about hadoop support, but OS-level (or maybe even network
card level) port bonding looks like the way to go.
On Fri, Aug 20, 2010 at 11:36 AM, Buttler, David <bu...@llnl.gov> wrote:
> Hi all,
> I have a small hbase cluster that I have recently filled with about 500M records (some of them quite large). One of the things that I notice when I do different types of map / reduce jobs over my table is that the network becomes a bottleneck. Currently I am running single gig Ethernet on this cluster, but it has 4 network ports.
>
> My question is this: is it possible to set up hadoop/hbase to take advantage of multiple networks connecting the computers?
> Could I specify multiple network connections in the config file?
> Would it make sense to put the region servers on a different network than the data nodes?
> Would it be more efficient to bond multiple channels at the OS level?
--
Alexey Kovyrin
http://kovyrin.net/