You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Arun Ramakrishnan <ar...@languageweaver.com> on 2010/07/08 01:09:00 UTC
rebalancing replciation help
Hi guys.
I have more than a specific question. I am going to layout the steps I have taken. Please comment on what I can do better.
I was trying to to add 5 nodes to my existing 10 node cluster and also increase the replication factor from 2 to 3.
I thought I don't have to run the balancer cause it would most likely put the new replicas into the new nodes.
There are about 500k blocks.
I wanted to get it all stabilized(replication and balancing) within 24 hours. Its more than 24 hours now and fsck reports 30% under replication. Is there a way to force hdfs to use balance/replicate more aggressively.
It would be great if someone explained what/when things happen to blocks in the context of
1) Rebalancing
2) -setrep
3) Restarting cluster with a higher/lower replication factor.
A few questions and a few issues here.
1) When you restart the cluster with a higher than previous replication value. Does it also apply to existing blocks or only to new blocks being created ?
2) Does the balancer take into account under replication of blocks or does it blindly start moving existing blocks to reach threshold ?
A very specific problem . I am having this strange problem where the -setrep hangs on one particular block for hours. Is this because its corrupt ?. But, fsck said its healthy.
Thanks
Arun
Re: rebalancing replciation help
Posted by Varene Olivier <va...@echo.fr>.
To answer your questions (adding to what alex already told you)
> 1) When you restart the cluster with a higher than previous
> replication value. Does it also apply to existing blocks or only to new
> blocks being created ?
it applies only to NEWLY created blocks
>
> 2) Does the balancer take into account under replication of blocks
> or does it blindly start moving existing blocks to reach threshold ?
i have no idea if the replication is performed before moving or not ...
should be the case for more security
Cheers
Arun Ramakrishnan a écrit :
> Hi guys.
>
> I have more than a specific question. I am going to layout the steps I
> have taken. Please comment on what I can do better.
>
>
>
> I was trying to to add 5 nodes to my existing 10 node cluster and also
> increase the replication factor from 2 to 3.
>
> I thought I don’t have to run the balancer cause it would most likely
> put the new replicas into the new nodes.
>
>
>
> There are about 500k blocks.
>
> I wanted to get it all stabilized(replication and balancing) within 24
> hours. Its more than 24 hours now and fsck reports 30% under
> replication. Is there a way to force hdfs to use balance/replicate more
> aggressively.
>
>
>
> It would be great if someone explained what/when things happen to blocks
> in the context of
>
> 1) Rebalancing
>
> 2) –setrep
>
> 3) Restarting cluster with a higher/lower replication factor.
>
>
>
> A few questions and a few issues here.
>
> 1) When you restart the cluster with a higher than previous
> replication value. Does it also apply to existing blocks or only to new
> blocks being created ?
>
> 2) Does the balancer take into account under replication of blocks
> or does it blindly start moving existing blocks to reach threshold ?
>
>
>
>
>
> A very specific problem . I am having this strange problem where the
> –setrep hangs on one particular block for hours. Is this because its
> corrupt ?. But, fsck said its healthy.
>
>
>
>
>
> Thanks
>
> Arun
>
RE: rebalancing replciation help
Posted by Arun Ramakrishnan <ar...@languageweaver.com>.
Thanks Alex.
From: Alex Loddengaard [mailto:alex@cloudera.com]
Sent: Thursday, July 08, 2010 11:39 AM
To: hdfs-user@hadoop.apache.org
Subject: Re: rebalancing replciation help
Hi Arun,
Consider setting dfs.balance.bandwidthPerSec to something as high as 20971520 for the balancer and the setrep. You can do this by supplying -D at the command line.
Your strategy for getting data onto the 5 nodes is correct: balance and setrep. Just understand these things take time.
Hope this helps.
Alex
On Wed, Jul 7, 2010 at 4:09 PM, Arun Ramakrishnan <ar...@languageweaver.com>> wrote:
Hi guys.
I have more than a specific question. I am going to layout the steps I have taken. Please comment on what I can do better.
I was trying to to add 5 nodes to my existing 10 node cluster and also increase the replication factor from 2 to 3.
I thought I don't have to run the balancer cause it would most likely put the new replicas into the new nodes.
There are about 500k blocks.
I wanted to get it all stabilized(replication and balancing) within 24 hours. Its more than 24 hours now and fsck reports 30% under replication. Is there a way to force hdfs to use balance/replicate more aggressively.
It would be great if someone explained what/when things happen to blocks in the context of
1) Rebalancing
2) -setrep
3) Restarting cluster with a higher/lower replication factor.
A few questions and a few issues here.
1) When you restart the cluster with a higher than previous replication value. Does it also apply to existing blocks or only to new blocks being created ?
2) Does the balancer take into account under replication of blocks or does it blindly start moving existing blocks to reach threshold ?
A very specific problem . I am having this strange problem where the -setrep hangs on one particular block for hours. Is this because its corrupt ?. But, fsck said its healthy.
Thanks
Arun
Re: rebalancing replciation help
Posted by Alex Loddengaard <al...@cloudera.com>.
Hi Arun,
Consider setting dfs.balance.bandwidthPerSec to something as high
as 20971520 for the balancer and the setrep. You can do this by supplying
-D at the command line.
Your strategy for getting data onto the 5 nodes is correct: balance and
setrep. Just understand these things take time.
Hope this helps.
Alex
On Wed, Jul 7, 2010 at 4:09 PM, Arun Ramakrishnan <
aramakrishnan@languageweaver.com> wrote:
> Hi guys.
>
> I have more than a specific question. I am going to layout the steps I
> have taken. Please comment on what I can do better.
>
>
>
> I was trying to to add 5 nodes to my existing 10 node cluster and also
> increase the replication factor from 2 to 3.
>
> I thought I don’t have to run the balancer cause it would most likely put
> the new replicas into the new nodes.
>
>
>
> There are about 500k blocks.
>
> I wanted to get it all stabilized(replication and balancing) within 24
> hours. Its more than 24 hours now and fsck reports 30% under replication. Is
> there a way to force hdfs to use balance/replicate more aggressively.
>
>
>
> It would be great if someone explained what/when things happen to blocks in
> the context of
>
> 1) Rebalancing
>
> 2) –setrep
>
> 3) Restarting cluster with a higher/lower replication factor.
>
>
>
> A few questions and a few issues here.
>
> 1) When you restart the cluster with a higher than previous
> replication value. Does it also apply to existing blocks or only to new
> blocks being created ?
>
> 2) Does the balancer take into account under replication of blocks or
> does it blindly start moving existing blocks to reach threshold ?
>
>
>
>
>
> A very specific problem . I am having this strange problem where the
> –setrep hangs on one particular block for hours. Is this because its corrupt
> ?. But, fsck said its healthy.
>
>
>
>
>
> Thanks
>
> Arun
>