slow transfer rates w/ HDFS balancer

[David Howell <de...@gmail.com>]

I just added a couple of new node to my cluster today, and ran the
balancer to get some data onto them. With scp, the transfer rate
between these machines is in the neighborhood of 50 MB/s... but the
transfers during the balancing cap out at 1 MB/s.

Is this to be expected, and if not, does anyone have any ideas of
where to start looking for a problem?

Thanks,
David

Re: slow transfer rates w/ HDFS balancer

[David Howell <de...@gmail.com>]

Awesome, that did the trick.

Thanks!

On Thu, Oct 8, 2009 at 2:06 PM, Allen Wittenauer
<aw...@linkedin.com> wrote:
>
>
>
> On 10/8/09 12:03 PM, "David Howell" <de...@gmail.com> wrote:
>
>> I just added a couple of new node to my cluster today, and ran the
>> balancer to get some data onto them. With scp, the transfer rate
>> between these machines is in the neighborhood of 50 MB/s... but the
>> transfers during the balancing cap out at 1 MB/s.
>>
>> Is this to be expected, and if not, does anyone have any ideas of
>> where to start looking for a problem?
>
> Balancer was intended to run as a background job so that the grid isn't
> limping while the balance is going on.  But this is a tunable: change the
> dfs.balance.bandwidthPerSec setting.
>
>

Re: slow transfer rates w/ HDFS balancer

[Allen Wittenauer <aw...@linkedin.com>]

On 10/8/09 12:03 PM, "David Howell" <de...@gmail.com> wrote:

> I just added a couple of new node to my cluster today, and ran the
> balancer to get some data onto them. With scp, the transfer rate
> between these machines is in the neighborhood of 50 MB/s... but the
> transfers during the balancing cap out at 1 MB/s.
> 
> Is this to be expected, and if not, does anyone have any ideas of
> where to start looking for a problem?

Balancer was intended to run as a background job so that the grid isn't
limping while the balance is going on.  But this is a tunable: change the
dfs.balance.bandwidthPerSec setting.