Stucked processed with older regionserver address assigned

[Buğra Çakır <bc...@beartell.com>]

Hi All,

There is an issue with HBase version 2.2.6 (bigtop).
We have difficulties running and finalizing Procedures/processes.

Each time we stopped gracefully and started regionserver and after
HBase Master. Below is the log..

We tried to change info:serverstartcode to the new regionserver but
unable to trigger the new regionserver accept the region.. So all
regions are in TRANSITION mode.


2022-07-06 11:31:52,019 WARN  [ProcExecTimeout]
assignment.AssignmentManager: STUCK Region-In-Transition rit=CLOSING,
location=bd09.xxx.local,16020,1654506758598, table=ALARM.Y_XXX_ALARM,
region=c4b5fdf7ccb355449113b686f22adfe0

region=c4b5fdf7ccb355449113b686f22adfe0, REOPEN/MOVE; rit=CLOSING,
location=bd09.xxx.local,16020,1654506758598; waiting on rectified
condition fixed by other Procedure or operator intervention
org.apache.hadoop.hbase.HBaseIOException: Failed to close region
	at
org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.c
onfirmClosed(TransitRegionStateProcedure.java:295)
	at
org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.e
xecuteFromState(TransitRegionStateProcedure.java:351)
	at
org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.e
xecuteFromState(TransitRegionStateProcedure.java:102)
	at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateM
achineProcedure.java:194)
	at
org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.e
xecute(TransitRegionStateProcedure.java:323)
	at
org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.e
xecute(TransitRegionStateProcedure.java:102)
	at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:9
62)
	at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(Proc
edureExecutor.java:1662)
	at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(P
rocedureExecutor.java:1409)
	at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(Proced
ureExecutor.java:78)
	at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(P
rocedureExecutor.java:1979)
2022-07-06 11:08:22,468 INFO  [PEWorker-1]
procedure2.TimeoutExecutorThread: ADDED pid=58534,
state=WAITING_TIMEOUT:REGION_STATE_TRANSITION_CLOSE, locked=true;
TransitRegionStateProcedure table=ALARM.Y_XXX_ALARM,
region=c4b5fdf7ccb355449113b686f22adfe0, REOPEN/MOVE; timeout=128000,
timestamp=1657095030468

2022-07-06 11:08:22,468 DEBUG [PEWorker-1]
procedure2.RootProcedureState: Add procedure pid=58534,
state=WAITING_TIMEOUT:REGION_STATE_TRANSITION_CLOSE, locked=true;
TransitRegionStateProcedure table=ALARM.Y_XXX_ALARM,
region=c4b5fdf7ccb355449113b686f22adfe0, REOPEN/MOVE as the 23th
rollback step