You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "anonymous (Jira)" <ji...@apache.org> on 2021/11/04 03:38:00 UTC
[jira] [Updated] (HBASE-26391) Unexpected crash in the process of
deleting a table causes stale data in ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-26391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
anonymous updated HBASE-26391:
------------------------------
Description:
There is a table {{mytable}} in a HBase cluster with two HMasters and two RegionServers.
1. Current active HMaster is HMaster1;
2. Client request to delete the table {{mytable}};
3. HMaster1 deletes znode in ZooKeeper: /hbase/table/mytable
{code:java}
[org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1367), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1359), org.apache.hadoop.hbase.zookeeper.ZKTableStateManager.setDeletedTable(ZKTableStateManager.java:244), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.deleteAssignmentState(DeleteTableProcedure.java:383), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:136), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:57), org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139), org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1166), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:954), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)]
{code}
4. HMaster1 crashes before deleting the znode: /hbase/table-lock/mytable
{code:java}
[org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1341), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1330), org.apache.hadoop.hbase.master.TableLockManager$ZKTableLockManager.tableDeleted(TableLockManager.java:442), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.markTableAsDeleted(MasterProcedureScheduler.java:895), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.completionCleanup(MasterProcedureScheduler.java:261), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.procedureFinished(ProcedureExecutor.java:1415), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:974), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)]
{code}
5. HMaster2 becomes active
6. The table {{mytablle}} was deleted, but the znode "/hbase/table-lock/mytable" was left on ZooKeeper
was:
There is a table {{mytable}} in a HBase cluster with two HMasters and two RegionServers.
1. Current active HMaster is HMaster1;
2. Client request to delete the table {{mytable}};
3. HMaster1 tries to delete znode in ZooKeeper: /hbase/table/mytable
{code:java}
[org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1367), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1359), org.apache.hadoop.hbase.zookeeper.ZKTableStateManager.setDeletedTable(ZKTableStateManager.java:244), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.deleteAssignmentState(DeleteTableProcedure.java:383), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:136), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:57), org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139), org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1166), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:954), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)]
{code}
4. HMaster1 crashes before deleting the znode: /hbase/table-lock/mytable
{code:java}
[org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1341), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1330), org.apache.hadoop.hbase.master.TableLockManager$ZKTableLockManager.tableDeleted(TableLockManager.java:442), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.markTableAsDeleted(MasterProcedureScheduler.java:895), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.completionCleanup(MasterProcedureScheduler.java:261), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.procedureFinished(ProcedureExecutor.java:1415), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:974), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)]
{code}
5. HMaster2 becomes active
6. The table {{mytablle}} was deleted, but the znode "/hbase/table-lock/mytable" was left on ZooKeeper
> Unexpected crash in the process of deleting a table causes stale data in ZooKeeper
> ----------------------------------------------------------------------------------
>
> Key: HBASE-26391
> URL: https://issues.apache.org/jira/browse/HBASE-26391
> Project: HBase
> Issue Type: Bug
> Affects Versions: 1.7.1
> Reporter: anonymous
> Priority: Major
>
> There is a table {{mytable}} in a HBase cluster with two HMasters and two RegionServers.
> 1. Current active HMaster is HMaster1;
> 2. Client request to delete the table {{mytable}};
> 3. HMaster1 deletes znode in ZooKeeper: /hbase/table/mytable
> {code:java}
> [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1367), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1359), org.apache.hadoop.hbase.zookeeper.ZKTableStateManager.setDeletedTable(ZKTableStateManager.java:244), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.deleteAssignmentState(DeleteTableProcedure.java:383), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:136), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:57), org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139), org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1166), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:954), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)]
> {code}
> 4. HMaster1 crashes before deleting the znode: /hbase/table-lock/mytable
> {code:java}
> [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1341), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1330), org.apache.hadoop.hbase.master.TableLockManager$ZKTableLockManager.tableDeleted(TableLockManager.java:442), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.markTableAsDeleted(MasterProcedureScheduler.java:895), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.completionCleanup(MasterProcedureScheduler.java:261), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.procedureFinished(ProcedureExecutor.java:1415), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:974), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)]
> {code}
> 5. HMaster2 becomes active
> 6. The table {{mytablle}} was deleted, but the znode "/hbase/table-lock/mytable" was left on ZooKeeper
--
This message was sent by Atlassian Jira
(v8.3.4#803005)