hbase:meta region got stuck in Opening state

[Chetan Rautela <ra...@gmail.com>]

Hello Members,

I am using Hbase 2.4.4 and facing some issues,

1: hbase:meta region is getting stuck at opening state while restarting
Hbase cluster.

Symptoms:

A few of the region servers and Master nodes went down, including one which
was holding the hbase:meta region.

After restarting, in logs I am seeing a holding pattern until region online
with Server Crash Procedure(SCP) as true.

I waited for 30 mins but the procedure was still stuck.

 Actions performed.

1: Bypass all stuck procedures using HBCK2.

2: Try to assign hbase:meta from HBCK2.  After this new procedure ID got
created but still it was stuck in the Opening state.

3: Try to set region state OPEN and CLOSED and restart master but no luck
on meta transition state.

Finally stopped all region servers and masters, then started only one
master and one region server, in which meta was  previously hosted. And
after assigning from HBCK2, meta got assigned successfully, further before
assigning hbase:namespace, started all other region servers and then
assigned namespace as well after this Hbase cluster came up.

Observation is that hbase:meta is only coming up in the same region server
where it was hosted before restart.

This is the first time I have seen this behaviour.

Seeking help if someone faced issues where the meta region is stuck in the
opening state, and how can I further debug why it is getting stuck in the
opening state .

Hbase region balancer was on during this activity.



Regards,

Chetan