Decommission of nodes

[Mat Hofschen <ho...@gmail.com>]

Hello,
I just needed to remove 8 machines from our 33 node cluster. (hadoop-0.19.1,
hbase-0.19.2)
The first 4 nodes I decommissioned in hadoop with the exclude-file and hbase
running. After 4 hours I gave up and stopped the machines by hand (they
never finished the decommissioning in progress phase).
For the next 4 nodes I stopped the hbase cluster then decommissioned the
nodes in hadoop. After 10 minutes the decommissioning finished successfully.


Is there a recommended way to remove nodes from hbase and how does that best
work together with decommissioning in hadoop.
How does hbase reconstruct the meta information during or after removal of
nodes?

Thanks for your help,
Matthias

Re: Decommission of nodes

[Billy Pearson <sa...@pearsonwholesale.com>]

The only reason I can thank of is the regionserver holding a open file on 
the datanodes like the log file for writing and the datanode is waiting for 
it close before moving it and exiting.

Billy


"Mat Hofschen" <ho...@gmail.com> wrote in 
message news:d2b99530905280412s1c18e22dkc2f9b981a5889f5e@mail.gmail.com...
> Hi,
>
> how does hbase react to hadoop decommissioning nodes?
> Should I shutdown the regionservers first, before decommissioning or does 
> it
> matter at all? Is it possible that a running hbase prevents 
> decommissioning
> to finish?
>
> Matthias
>
>
>
> On Wed, May 27, 2009 at 7:44 PM, Billy Pearson
> <sa...@pearsonwholesale.com>wrote:
>
>> the decommissioning process in hadoop takes a little while I thank there
>> the balancing bandwidth has a lot to do with it.
>> but to stop region server you should be able to run
>> bin/hbase-daemon.sh stop regionserver
>> on the server you want to stop the hbase as long as it not the master 
>> then
>> you can do this on a live cluster.
>> as each goes down they will tell the master by closing the regions and
>> releasing its leases and the master will start to redeploy the regions to
>> other servers.
>>
>> Billy
>>
>>
>> "Mat Hofschen" <ho...@gmail.com> wrote 
>> in message
>> news:d2b99530905270754p566be51n8dc468f9d1d2384b@mail.gmail.com...
>>
>>  Hello,
>>> I just needed to remove 8 machines from our 33 node cluster.
>>> (hadoop-0.19.1,
>>> hbase-0.19.2)
>>> The first 4 nodes I decommissioned in hadoop with the exclude-file and
>>> hbase
>>> running. After 4 hours I gave up and stopped the machines by hand (they
>>> never finished the decommissioning in progress phase).
>>> For the next 4 nodes I stopped the hbase cluster then decommissioned the
>>> nodes in hadoop. After 10 minutes the decommissioning finished
>>> successfully.
>>>
>>>
>>> Is there a recommended way to remove nodes from hbase and how does that
>>> best
>>> work together with decommissioning in hadoop.
>>> How does hbase reconstruct the meta information during or after removal 
>>> of
>>> nodes?
>>>
>>> Thanks for your help,
>>> Matthias
>>>
>>>
>>
>>
> 

Re: Decommission of nodes

[stack <st...@duboce.net>]

On Thu, May 28, 2009 at 4:12 AM, Mat Hofschen <ho...@gmail.com> wrote:
> Hi,
>
> how does hbase react to hadoop decommissioning nodes?

HBase is like a fish out of water if you yank its hdfs out from under
it (it should 'adjust' if you are just pulling datanodes in a manner
that does not damage block replicas though we are not as robust as we
should be because of https://issues.apache.org/jira/browse/HADOOP-4681
-- if you are going to be doing lots of decommissioning, then would
suggest you appy this patch to your hadoop)

> Should I shutdown the regionservers first, before decommissioning or does it
> matter at all? Is it possible that a running hbase prevents decommissioning
> to finish?

Well, regionservers are getting blocks from all over hdfs, they are
not bound to any individual datanode but probably best to take down
the regionservers you'd like to decommission first to minimize the set
of servers that need to adjust to the new hdfs topography.

St.Ack


>
> Matthias
>
>
>
> On Wed, May 27, 2009 at 7:44 PM, Billy Pearson
> <sa...@pearsonwholesale.com>wrote:
>
>> the decommissioning process in hadoop takes a little while I thank there
>> the balancing bandwidth has a lot to do with it.
>> but to stop region server you should be able to run
>> bin/hbase-daemon.sh stop regionserver
>> on the server you want to stop the hbase as long as it not the master then
>> you can do this on a live cluster.
>> as each goes down they will tell the master by closing the regions and
>> releasing its leases and the master will start to redeploy the regions to
>> other servers.
>>
>> Billy
>>
>>
>> "Mat Hofschen" <ho...@gmail.com> wrote in message
>> news:d2b99530905270754p566be51n8dc468f9d1d2384b@mail.gmail.com...
>>
>>  Hello,
>>> I just needed to remove 8 machines from our 33 node cluster.
>>> (hadoop-0.19.1,
>>> hbase-0.19.2)
>>> The first 4 nodes I decommissioned in hadoop with the exclude-file and
>>> hbase
>>> running. After 4 hours I gave up and stopped the machines by hand (they
>>> never finished the decommissioning in progress phase).
>>> For the next 4 nodes I stopped the hbase cluster then decommissioned the
>>> nodes in hadoop. After 10 minutes the decommissioning finished
>>> successfully.
>>>
>>>
>>> Is there a recommended way to remove nodes from hbase and how does that
>>> best
>>> work together with decommissioning in hadoop.
>>> How does hbase reconstruct the meta information during or after removal of
>>> nodes?
>>>
>>> Thanks for your help,
>>> Matthias
>>>
>>>
>>
>>
>

Re: Decommission of nodes

[Mat Hofschen <ho...@gmail.com>]

Hi,

how does hbase react to hadoop decommissioning nodes?
Should I shutdown the regionservers first, before decommissioning or does it
matter at all? Is it possible that a running hbase prevents decommissioning
to finish?

Matthias



On Wed, May 27, 2009 at 7:44 PM, Billy Pearson
<sa...@pearsonwholesale.com>wrote:

> the decommissioning process in hadoop takes a little while I thank there
> the balancing bandwidth has a lot to do with it.
> but to stop region server you should be able to run
> bin/hbase-daemon.sh stop regionserver
> on the server you want to stop the hbase as long as it not the master then
> you can do this on a live cluster.
> as each goes down they will tell the master by closing the regions and
> releasing its leases and the master will start to redeploy the regions to
> other servers.
>
> Billy
>
>
> "Mat Hofschen" <ho...@gmail.com> wrote in message
> news:d2b99530905270754p566be51n8dc468f9d1d2384b@mail.gmail.com...
>
>  Hello,
>> I just needed to remove 8 machines from our 33 node cluster.
>> (hadoop-0.19.1,
>> hbase-0.19.2)
>> The first 4 nodes I decommissioned in hadoop with the exclude-file and
>> hbase
>> running. After 4 hours I gave up and stopped the machines by hand (they
>> never finished the decommissioning in progress phase).
>> For the next 4 nodes I stopped the hbase cluster then decommissioned the
>> nodes in hadoop. After 10 minutes the decommissioning finished
>> successfully.
>>
>>
>> Is there a recommended way to remove nodes from hbase and how does that
>> best
>> work together with decommissioning in hadoop.
>> How does hbase reconstruct the meta information during or after removal of
>> nodes?
>>
>> Thanks for your help,
>> Matthias
>>
>>
>
>

Re: Decommission of nodes

[Billy Pearson <sa...@pearsonwholesale.com>]

the decommissioning process in hadoop takes a little while I thank there the 
balancing bandwidth has a lot to do with it.
but to stop region server you should be able to run
bin/hbase-daemon.sh stop regionserver
on the server you want to stop the hbase as long as it not the master then 
you can do this on a live cluster.
as each goes down they will tell the master by closing the regions and 
releasing its leases and the master will start to redeploy the regions to 
other servers.

Billy


"Mat Hofschen" <ho...@gmail.com> wrote in 
message news:d2b99530905270754p566be51n8dc468f9d1d2384b@mail.gmail.com...
> Hello,
> I just needed to remove 8 machines from our 33 node cluster. 
> (hadoop-0.19.1,
> hbase-0.19.2)
> The first 4 nodes I decommissioned in hadoop with the exclude-file and 
> hbase
> running. After 4 hours I gave up and stopped the machines by hand (they
> never finished the decommissioning in progress phase).
> For the next 4 nodes I stopped the hbase cluster then decommissioned the
> nodes in hadoop. After 10 minutes the decommissioning finished 
> successfully.
>
>
> Is there a recommended way to remove nodes from hbase and how does that 
> best
> work together with decommissioning in hadoop.
> How does hbase reconstruct the meta information during or after removal of
> nodes?
>
> Thanks for your help,
> Matthias
>