corrupted procedures

[Hamado Dene <ha...@yahoo.com.INVALID>]

Hi,
I am having which problem after upgrading hbase from 2.2.6 to hbase 2.3.7.When I try to reboot the master, the reboot fails with the following error:
I am having which problem after upgrading hbase from 2.2.6 to hbase 2.3.7.When I try to reboot the master, the reboot fails with the error below.The offending table appears to be usertable. But I didn't find instructions on how to fix this. table.Can anyone help me fix this problem?


 022-07-07 11:21:42,378 WARN  [master/vhadoop2-hn01:16000:becomeActiveMaster] wal.ProcedureWALFormatReader: Nothing left to decode. Exiting with missing EOF, log=hdfs://hadoopcluster/hbase/MasterProcWALs/pv2-00000000000000000273.log2022-07-07 11:21:42,378 INFO  [master/vhadoop2-hn01:16000:becomeActiveMaster] wal.ProcedureWALFormatReader: Read 1 entries in hdfs://hadoopcluster/hbase/MasterProcWALs/pv2-00000000000000000273.log2022-07-07 11:21:42,393 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] store.ProcedureTree: Missing stack id 328, max stack id is 341, root procedure is Procedure(pid=7811, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure)2022-07-07 11:21:42,393 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] store.ProcedureTree: Missing stack id 340, max stack id is 341, root procedure is Procedure(pid=7811, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure)2022-07-07 11:21:42,395 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] store.ProcedureTree: Orphan procedure: Procedure(pid=19, ppid=8, class=org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure)2022-07-07 11:21:42,456 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=19, ppid=8, state=WAITING_TIMEOUT; OpenRegionProcedure b2ed5d26fc3940adb3bc313ab0fec303, server=vhadoop2-hd01.sviluppo.dna,16020,16155544155642022-07-07 11:21:42,492 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7811, state=WAITING:DISABLE_TABLE_ADD_REPLICATION_BARRIER; DisableTableProcedure table=usertable2022-07-07 11:21:42,512 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7812, ppid=7811, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=a42c2943c9dd4a2e3177ee3a9504d63b, UNASSIGN2022-07-07 11:21:42,512 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7813, ppid=7811, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=7631bb1a4c44d97eafaa6c80f75d6e82, UNASSIGN2022-07-07 11:21:42,513 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7814, ppid=7811, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=b0220c4eca1c97e563531f8cd34787fc, UNASSIGN2022-07-07 11:21:42,515 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7815, ppid=7811, state=SUCCESS; TransitRegionStateProcedure table=usertable, region=5f081ee337e4fc11fa503f4e1d4084b4, UNASSIGN2022-07-07 11:21:42,515 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7816, ppid=7811, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=66870df55b6e72315f96d3c6951eb4ff, UNASSIGN2022-07-07 11:21:42,516 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7817, ppid=7811, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=e2043a2cf0b658c02ab9c55fc22622f5, UNASSIGN2022-07-07 11:21:42,517 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7818, ppid=7811, state=SUCCESS; TransitRegionStateProcedure table=usertable, region=cc7bea6e52f753a194e675124339c43d, UNASSIGN2022-07-07 11:21:42,518 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7819, ppid=7811, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=84ae5ec65a3f6c0d8ffd2dfc8c6c831c, UNASSIGN2022-07-07 11:21:42,519 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7820, ppid=7811, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=8777a06ef3d0edfe2058b3e505f8a530, UNASSIGN2022-07-07 11:21:42,519 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7821, ppid=7811, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=868ce5a20c07134c88f68f892c0f82e6, UNASSIGN2022-07-07 11:21:42,519 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7822, ppid=7811, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=9a17e6bc5bf9cc4fe5b6de9427f97f8c, UNASSIGN2022-07-07 11:21:42,520 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7823, ppid=7811, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=3e437e8e36d823990d90ee7e0d2ebef1, UNASSIGN2022-07-07 11:21:42,520 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7824, ppid=7811, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=423f311ed32c0066d145000c5e303ea8, UNASSIGN2022-07-07 11:21:42,520 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7825, ppid=7811, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=usertable, region=e50b42258f44df29b64fc7d09bf3341c, UNASSIGN2022-07-07 11:21:42,521 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] region.RegionProcedureStore: Corrupted procedure pid=7826, ppid=7811, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionState

2022-07-07 11:21:42,609 ERROR [master/vhadoop2-hn01:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.io.IOException: There are 154 corrupted procedures when migrating from the old WAL based store to the new region based store, please fix them before upgrading again.        at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore$1.handleCorrupted(RegionProcedureStore.java:210)        at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$2.handleCorrupted(WALProcedureStore.java:479)        at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.finish(ProcedureWALFormatReader.java:152)        at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:103)        at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:465)        at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.tryMigrate(RegionProcedureStore.java:184)        at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.recoverLease(RegionProcedureStore.java:257)        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:587)        at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1666)        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:961)        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2336)        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626)        at java.lang.Thread.run(Thread.java:748)

Thanks in advance,
Hamado Dene