[jira] [Created] (HBASE-27520) The potential delay of HDFS RPC in HRegion may cause data inconsistency

["Haoze Wu (Jira)" <ji...@apache.org>]

Haoze Wu created HBASE-27520:
--------------------------------

             Summary: The potential delay of HDFS RPC in HRegion may cause data inconsistency
                 Key: HBASE-27520
                 URL: https://issues.apache.org/jira/browse/HBASE-27520
             Project: HBase
          Issue Type: Bug
    Affects Versions: 2.4.2
            Reporter: Haoze Wu


This is a follow-up for HBASE-26256 about the data inconsistency issue.

The data inconsistency issue is that when the HBase shell is running the `create` command, the metadata is not finalized or committed until the shell command returns, but other clients can see the table “created” when they use the “list” command. We recently did more analysis and found the root cause.

Here is the workflow. When a table is created, the CreateTableProcedure is created at HMaster#createTable at line 2011:

 
{code:java}
return MasterProcedureUtil
 .submitProcedure(new MasterProcedureUtil.NonceProcedureRunnable(this, nonceGroup, nonce) {
   @Override
   protected void run() throws IOException {
     getMaster().getMasterCoprocessorHost().preCreateTable(desc, newRegions);

     LOG.info(getClientIdAuditPrefix() + " create " + desc); 

     // TODO: We can handle/merge duplicate requests, and differentiate the case of
     // TableExistsException by saying if the schema is the same or not.
     //
     // We need to wait for the procedure to potentially fail due to "prepare" sanity
     // checks. This will block only the beginning of the procedure. See HBASE-19953.
     ProcedurePrepareLatch latch = ProcedurePrepareLatch.createBlockingLatch();
     submitProcedure(
       new CreateTableProcedure(procedureExecutor.getEnvironment(), desc, newRegions, latch));             // line 2011
     latch.await();

     getMaster().getMasterCoprocessorHost().postCreateTable(desc, newRegions);
   }

   @Override
   protected String getDescription() {
     return "CreateTableProcedure";
   }
 });{code}
The its call stack is:

 

 
{code:java}
org.apache.hadoop.hbase.master.HMaster,createTable,1995
org.apache.hadoop.hbase.master.MasterRpcServices,createTable,705
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2,callBlockingMethod,108919
org.apache.hadoop.hbase.ipc.RpcServer,call,395
org.apache.hadoop.hbase.ipc.CallRunner,run,133
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler,run,338
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler,run,318 {code}
`CreateTableProcedure` is submitted at line 2011. It will eventually call CreateTableProcedure#executeFromState in HMaster.

 

CreateTableProcedure#executeFromState runs a state machine. It exercises 3 states, one after another (via the loop in ProcedureExecutor#executeProcedure):
 # `CREATE_TABLE_ADD_TO_META` state. It will add table info to the hbase::meta.
 # `CREATE_TABLE_ASSIGN_REGIONS` state. It will create a subprocedure, which makes an AssignmentManager in master connect to regionservers to distribute the region create command.
 # `CREATE_TABLE_UPDATE_DESC_CACHE` state. It finalizes the region creation.

Code snippet (CreateTableProcedure#executeFromState, line 105 ~ 120):

 
{code:java}
case CREATE_TABLE_ADD_TO_META:
 newRegions = addTableToMeta(env, tableDescriptor, newRegions);
 setNextState(CreateTableState.CREATE_TABLE_ASSIGN_REGIONS);
 break;
case CREATE_TABLE_ASSIGN_REGIONS:
 setEnablingState(env, getTableName());
 addChildProcedure(env.getAssignmentManager()
   .createRoundRobinAssignProcedures(newRegions));
 setNextState(CreateTableState.CREATE_TABLE_UPDATE_DESC_CACHE);
 break;
case CREATE_TABLE_UPDATE_DESC_CACHE:
 // XXX: this stage should be named as set table enabled, as now we will cache the
 // descriptor after writing fs layout.
 setEnabledState(env, getTableName());
 setNextState(CreateTableState.CREATE_TABLE_POST_OPERATION);
 break;{code}
When the issue described in HBASE-26256 appears, it is in the `CREATE_TABLE_ASSIGN_REGIONS` state. The subprocedure calling regionserver is stuck for a while, so the state machine is not able to proceed to the `CREATE_TABLE_UPDATE_DESC_CACHE` state temporarily. However, at this moment, when other clients run the `list` command, they are essentially only checking the meta variable field in this HMaster and print this table, although the state machine here is still stuck at the `CREATE_TABLE_ASSIGN_REGIONS` and this table is actually not yet fully created. Therefore, the inconsistency happens.

To fix this issue, we need to slightly change the state machine or add some auxiliary flags for the tables to mark whether the table is still under the procedure of being created. We think the latter is better because changing the state machine may involve lots of modification.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)