best way to replace disks on a small cluster

[Marco Cadetg <ma...@zattoo.com>]

Hi there,

Current situation:
3 slaves with each two 320GB disks in RAID 1. All the disks show high read
errors and io throughput has gone below 5Mb/s without running any hadoop
job. (It looks like it will fall apart soon...)

What is the best way to replace the bad disks? I may be able to add another
two machines into the mix. I can't / won't rebuild the RAID as my new disks
will be 2TB each, so I wouldn't like to use only 320GB of them.

Is the best way to add two new nodes into the mix and then mark two other
machines to dfs.host.exclude. And after some time I can take them out???

Thanks for your help,
-Marco

Re: best way to replace disks on a small cluster

[Allen Wittenauer <aw...@apache.org>]

On Sep 7, 2011, at 6:19 AM, Marco Cadetg wrote:
> Current situation:
> 3 slaves with each two 320GB disks in RAID 1. All the disks show high read
> errors and io throughput has gone below 5Mb/s without running any hadoop
> job. (It looks like it will fall apart soon...)

	 One the special characteristics of RAID1 (and some implementations of other striping layouts) is that the speed of the array is (essentially) the speed of the slowest disk.  When drives are healthy, this speed is tremendous.  as soon as one disk starts to head into troubled territories, massive performance drops are typical.  Given Hadoop incredible ability to grind drives into dust, this tends to happen more often than not. :)  This is one of the reasons why RAID isn't generally recommended for data nodes.

> What is the best way to replace the bad disks? I may be able to add another
> two machines into the mix. I can't / won't rebuild the RAID as my new disks
> will be 2TB each, so I wouldn't like to use only 320GB of them.
> 
> Is the best way to add two new nodes into the mix and then mark two other
> machines to dfs.host.exclude. And after some time I can take them out???

	That sounds reasonable with only three slaves.  Just make sure that fsck comes back clean and that all files have a rep factor of 3.  Another choice is to just replace one machine's drives, wait till fsck comes back clean, rinse/repeat.  

Re: best way to replace disks on a small cluster

[Marco Cadetg <ma...@zattoo.com>]

Hi Mike,

Thanks for your quick reply.

On Wed, Sep 7, 2011 at 3:21 PM, Segel, Mike <ms...@navteq.com> wrote:

> Hi,
>
> Ok, so you have a small cluster w raid drives...
> First, raid isn't really a good idea.
>
> You want to replace the old raid drives w new 2TB drives in JBOD.
> (Which is better.)
>

I'll do this then when I replace the disks.


>
> What percentage of your drives are filled?
>

The percentage is 53% on all nodes, so it should theoretically be possible
to do. On the other hand I may be able to add another node into the mix
which by then means that it will work.

Thanks,
-Marco


> Assuming you have enough space on 2 of your DNs, you can decommission one,
> run fsck and balancer to compensate for the lost blocks, then rebuild your
> node.
> Replace the drives w the 2TB as JBOD drives, bring the node back up.
> Rebalance the cluster to get more of the data on the machine, and repeat
> the process for the other nodes.
>
> HTH
>
> -Mike
>
> ________________________________________
> From: Marco Cadetg [marco@zattoo.com]
> Sent: Wednesday, September 07, 2011 8:19 AM
> To: general@hadoop.apache.org
> Subject: best way to replace disks on a small cluster
>
> Hi there,
>
> Current situation:
> 3 slaves with each two 320GB disks in RAID 1. All the disks show high read
> errors and io throughput has gone below 5Mb/s without running any hadoop
> job. (It looks like it will fall apart soon...)
>
> What is the best way to replace the bad disks? I may be able to add another
> two machines into the mix. I can't / won't rebuild the RAID as my new disks
> will be 2TB each, so I wouldn't like to use only 320GB of them.
>
> Is the best way to add two new nodes into the mix and then mark two other
> machines to dfs.host.exclude. And after some time I can take them out???
>
> Thanks for your help,
> -Marco
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above.  If you are not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>

RE: best way to replace disks on a small cluster

["Segel, Mike" <ms...@navteq.com>]

Hi,

Ok, so you have a small cluster w raid drives...
First, raid isn't really a good idea.

You want to replace the old raid drives w new 2TB drives in JBOD.
(Which is better.)

What percentage of your drives are filled?
Assuming you have enough space on 2 of your DNs, you can decommission one, run fsck and balancer to compensate for the lost blocks, then rebuild your node.
Replace the drives w the 2TB as JBOD drives, bring the node back up.
Rebalance the cluster to get more of the data on the machine, and repeat the process for the other nodes.

HTH

-Mike

________________________________________
From: Marco Cadetg [marco@zattoo.com]
Sent: Wednesday, September 07, 2011 8:19 AM
To: general@hadoop.apache.org
Subject: best way to replace disks on a small cluster

Hi there,

Current situation:
3 slaves with each two 320GB disks in RAID 1. All the disks show high read
errors and io throughput has gone below 5Mb/s without running any hadoop
job. (It looks like it will fall apart soon...)

What is the best way to replace the bad disks? I may be able to add another
two machines into the mix. I can't / won't rebuild the RAID as my new disks
will be 2TB each, so I wouldn't like to use only 320GB of them.

Is the best way to add two new nodes into the mix and then mark two other
machines to dfs.host.exclude. And after some time I can take them out???

Thanks for your help,
-Marco

The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.