What are the appropriate steps before performing hardware maintenance?

[Dan Crosta <da...@magnetic.com>]

We have to perform maintenance on one of our HDFS DataNode/HBase Regionserver machines for a few hours. What are the right steps to take before doing the maintenance in order to ensure limited  impact to the cluster and (thrift) clients of the cluster, both for HDFS and HBase?

After the maintenance, are there any special steps required to add the node back to the cluster, or can we simply restart the services and HDFS/HBase take care of the rest?

Thanks,
- Dan

Re: What are the appropriate steps before performing hardware maintenance?

[Jean-Marc Spaggiari <je...@spaggiari.org>]

Moving to scm-users@cloudera.org then (hbase in BCC).

Hi Dan,

The best way to know how to achieve this with Cloudera Manager is to
ask on the scm-users list.

I'm net yet enough used to CM to reply to your question so I will let
someone else confirm.

JMS



2013/4/25 Dan Crosta <da...@magnetic.com>:
> Sorry, I should have mentioned before -- we are using CDH 4.2, which does not package the graceful_stop script. Do you happen to know if there's a way to do this through the CDH manager? Perhaps the "decommission" action does something similar? My impression is that "decommission" is more heavy-handed, but if that's the most convenient route, that'll work for us.
>
> Thanks,
> - Dan
>
> On Apr 25, 2013, at 11:30 AM, Jean-Marc Spaggiari wrote:
>
>> Hi Dan,
>>
>> You might want to take a look at bin/graceful_stop.sh . It will move
>> all the regions hosted by your RS to other RS before stopping it
>> gracefuly. After the maintenance, simply start the RS/DN back and it
>> will be added back to the cluster. Loadbalancer will then assign some
>> regions back to him. You will loose some data locality for the regions
>> wich are going to be moved.
>>
>> JM
>>
>> 2013/4/25 Dan Crosta <da...@magnetic.com>:
>>> We have to perform maintenance on one of our HDFS DataNode/HBase Regionserver machines for a few hours. What are the right steps to take before doing the maintenance in order to ensure limited  impact to the cluster and (thrift) clients of the cluster, both for HDFS and HBase?
>>>
>>> After the maintenance, are there any special steps required to add the node back to the cluster, or can we simply restart the services and HDFS/HBase take care of the rest?
>>>
>>> Thanks,
>>> - Dan
>

Re: What are the appropriate steps before performing hardware maintenance?

[Dan Crosta <da...@magnetic.com>]

Sorry, I should have mentioned before -- we are using CDH 4.2, which does not package the graceful_stop script. Do you happen to know if there's a way to do this through the CDH manager? Perhaps the "decommission" action does something similar? My impression is that "decommission" is more heavy-handed, but if that's the most convenient route, that'll work for us.

Thanks,
- Dan

On Apr 25, 2013, at 11:30 AM, Jean-Marc Spaggiari wrote:

> Hi Dan,
> 
> You might want to take a look at bin/graceful_stop.sh . It will move
> all the regions hosted by your RS to other RS before stopping it
> gracefuly. After the maintenance, simply start the RS/DN back and it
> will be added back to the cluster. Loadbalancer will then assign some
> regions back to him. You will loose some data locality for the regions
> wich are going to be moved.
> 
> JM
> 
> 2013/4/25 Dan Crosta <da...@magnetic.com>:
>> We have to perform maintenance on one of our HDFS DataNode/HBase Regionserver machines for a few hours. What are the right steps to take before doing the maintenance in order to ensure limited  impact to the cluster and (thrift) clients of the cluster, both for HDFS and HBase?
>> 
>> After the maintenance, are there any special steps required to add the node back to the cluster, or can we simply restart the services and HDFS/HBase take care of the rest?
>> 
>> Thanks,
>> - Dan

Re: What are the appropriate steps before performing hardware maintenance?

[Jean-Marc Spaggiari <je...@spaggiari.org>]

Hi Dan,

You might want to take a look at bin/graceful_stop.sh . It will move
all the regions hosted by your RS to other RS before stopping it
gracefuly. After the maintenance, simply start the RS/DN back and it
will be added back to the cluster. Loadbalancer will then assign some
regions back to him. You will loose some data locality for the regions
wich are going to be moved.

JM

2013/4/25 Dan Crosta <da...@magnetic.com>:
> We have to perform maintenance on one of our HDFS DataNode/HBase Regionserver machines for a few hours. What are the right steps to take before doing the maintenance in order to ensure limited  impact to the cluster and (thrift) clients of the cluster, both for HDFS and HBase?
>
> After the maintenance, are there any special steps required to add the node back to the cluster, or can we simply restart the services and HDFS/HBase take care of the rest?
>
> Thanks,
> - Dan