HMaster shows decommisioned servers as Dead Region Servers

[Srikanth Rajan <rs...@gmail.com>]

Hi Folks,

HBase version : 0.94.20

Recently we decommissioned some RegionServer hosts from HBase cluster. 
We cleaned up all hadoop/hbase configs including the "regionservers" 
file. Tried restarting HMaster Service and even entire cluster multiple 
times. But somehow HMaster is showing some of the decommissioned servers 
as "Dead Region Servers".

I could not find any traces of the decommissioned hostnames in 
Zookeeper. Also I did a full grep on hadoop/hbase installation 
directory, no traces of decommissioned hostnames there as well. I am 
just wondering where the HMaster is picking up the region server list from.

Any ideas ?

Thanks,
Srikanth

Re: HMaster shows decommisioned servers as Dead Region Servers

[Esteban Gutierrez <es...@cloudera.com>]

Hello Srikanth,

Could you please tell us more about the decommissioning process? Are you
using the hbase/bin/region_mover.rb script and then after completion are
you shutting down Region Server Java process? do you have any HBase master
log that you could upload to pastebin around the time you decommissioned
the RS and restarted the master?

cheers,
esteban.


--
Cloudera, Inc.



On Mon, Jul 28, 2014 at 3:16 AM, Srikanth Rajan <rs...@gmail.com>
wrote:

> Hi Folks,
>
> HBase version : 0.94.20
>
> Recently we decommissioned some RegionServer hosts from HBase cluster. We
> cleaned up all hadoop/hbase configs including the "regionservers" file.
> Tried restarting HMaster Service and even entire cluster multiple times.
> But somehow HMaster is showing some of the decommissioned servers as "Dead
> Region Servers".
>
> I could not find any traces of the decommissioned hostnames in Zookeeper.
> Also I did a full grep on hadoop/hbase installation directory, no traces of
> decommissioned hostnames there as well. I am just wondering where the
> HMaster is picking up the region server list from.
>
> Any ideas ?
>
> Thanks,
> Srikanth
>
>