[jira] [Commented] (HBASE-22608) MVCC's writeEntry didn't complete and make MVCC stuck

["Zheng Hu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-22608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878382#comment-16878382 ] 

Zheng Hu commented on HBASE-22608:
----------------------------------

Have not found the way to reproduce this bug now,  plan to add some debug log in the compacting memstore, and deploy it on our staging cluster to see what happened.

> MVCC's writeEntry didn't complete and make MVCC stuck
> -----------------------------------------------------
>
>                 Key: HBASE-22608
>                 URL: https://issues.apache.org/jira/browse/HBASE-22608
>             Project: HBase
>          Issue Type: Bug
>          Components: in-memory-compaction
>            Reporter: Guanghao Zhang
>            Assignee: Zheng Hu
>            Priority: Critical
>
> {code:java}
> 2019-06-20,05:03:44,917 ERROR [RpcServer.default.RWQ.Fifo.write.handler=61,queue=1,port=22600] org.apache.hadoop.hbase.regionserver.HRegion: Asked to modify this region's (xxxxxx,,1560481375170.10b01c12d58ce75c9aaf1ac15cc2a7f3.) memStoreSizing to a negative value which is incorrect. Current memStoreSizing=-1686222, delta=1489930
> java.lang.Exception
> at org.apache.hadoop.hbase.regionserver.HRegion.checkNegativeMemStoreDataSize(HRegion.java:1317)
> at org.apache.hadoop.hbase.regionserver.HRegion.incMemStoreSize(HRegion.java:1295)
> at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3316)
> at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3821)
> at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4248)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4179)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4109)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1059)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:991)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:954)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2833)
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> {code}
> See [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3728]
> {code:java}
> @Override
> public WriteEntry writeMiniBatchOperationsToMemStore(
>     final MiniBatchOperationInProgress<Mutation> miniBatchOp, @Nullable WriteEntry writeEntry)
>     throws IOException {
>   if (writeEntry == null) {
>     writeEntry = region.mvcc.begin();
>   }
>   super.writeMiniBatchOperationsToMemStore(miniBatchOp, writeEntry.getWriteNumber());
>   return writeEntry;
> }
> {code}
> super.writeMiniBatchOperationsToMemStore throw a exception and the new writeEntry cannot be complete and make the MVCC stuck.
>  
> And we meet this problem when enable in-memory compaction. But that should be another issue and need to dig more.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)