You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Guanghao Zhang (JIRA)" <ji...@apache.org> on 2019/06/20 06:27:00 UTC
[jira] [Comment Edited] (HBASE-22608) MVCC's writeEntry didn't
complete and make MVCC stuck
[ https://issues.apache.org/jira/browse/HBASE-22608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868288#comment-16868288 ]
Guanghao Zhang edited comment on HBASE-22608 at 6/20/19 6:26 AM:
-----------------------------------------------------------------
Ok. This should be a problem of in memory compaction. Most handlers stuck on Segment.sharedLock. This is an RW lock. But most write handler cannot get the read lock. MVCC stuck because these batch-mutate operations cannot complete.
{code:java}
"RpcServer.default.RWQ.Fifo.write.handler=239,queue=3,port=22600" #330 daemon prio=5 os_prio=0 tid=0x00007fe43ece3ab0 nid=0x27966 runnable [0x00007fdbd3bac000]
java.lang.Thread.State: RUNNABLE
at java.lang.ThreadLocal$ThreadLocalMap.cleanSomeSlots(ThreadLocal.java:661)
at java.lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:483)
at java.lang.ThreadLocal$ThreadLocalMap.access$100(ThreadLocal.java:298)
at java.lang.ThreadLocal.set(ThreadLocal.java:203)
at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReadLock(ReentrantReadWriteLock.java:597)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:799)
at org.apache.hadoop.hbase.regionserver.Segment.sharedLock(Segment.java:253)
at org.apache.hadoop.hbase.regionserver.CompactingMemStore.preUpdate(CompactingMemStore.java:318)
at org.apache.hadoop.hbase.regionserver.AbstractMemStore.doAddOrUpsert(AbstractMemStore.java:145)
at org.apache.hadoop.hbase.regionserver.AbstractMemStore.add(AbstractMemStore.java:118)
at org.apache.hadoop.hbase.regionserver.AbstractMemStore.add(AbstractMemStore.java:112)
at org.apache.hadoop.hbase.regionserver.HStore.add(HStore.java:770)
at org.apache.hadoop.hbase.regionserver.HRegion.applyToMemStore(HRegion.java:4628)
at org.apache.hadoop.hbase.regionserver.HRegion.access$500(HRegion.java:232)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.applyFamilyMapToMemStore(HRegion.java:3621)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lambda$writeMiniBatchOperationsToMemStore$0(HRegion.java:3311)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation$$Lambda$475/1305519523.visit(Unknown Source)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations(HRegion.java:3240)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3303)
at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3821)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4248)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4179)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4109)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1059)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:991)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:954)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2833)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
{code}
was (Author: zghaobac):
Ok. This should be a problem of in memory compaction. Most handlers stuck on Segment.sharedLock. This is an RW lock. But most write handler cannot get the read lock.
{code:java}
"RpcServer.default.RWQ.Fifo.write.handler=239,queue=3,port=22600" #330 daemon prio=5 os_prio=0 tid=0x00007fe43ece3ab0 nid=0x27966 runnable [0x00007fdbd3bac000]
java.lang.Thread.State: RUNNABLE
at java.lang.ThreadLocal$ThreadLocalMap.cleanSomeSlots(ThreadLocal.java:661)
at java.lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:483)
at java.lang.ThreadLocal$ThreadLocalMap.access$100(ThreadLocal.java:298)
at java.lang.ThreadLocal.set(ThreadLocal.java:203)
at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReadLock(ReentrantReadWriteLock.java:597)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:799)
at org.apache.hadoop.hbase.regionserver.Segment.sharedLock(Segment.java:253)
at org.apache.hadoop.hbase.regionserver.CompactingMemStore.preUpdate(CompactingMemStore.java:318)
at org.apache.hadoop.hbase.regionserver.AbstractMemStore.doAddOrUpsert(AbstractMemStore.java:145)
at org.apache.hadoop.hbase.regionserver.AbstractMemStore.add(AbstractMemStore.java:118)
at org.apache.hadoop.hbase.regionserver.AbstractMemStore.add(AbstractMemStore.java:112)
at org.apache.hadoop.hbase.regionserver.HStore.add(HStore.java:770)
at org.apache.hadoop.hbase.regionserver.HRegion.applyToMemStore(HRegion.java:4628)
at org.apache.hadoop.hbase.regionserver.HRegion.access$500(HRegion.java:232)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.applyFamilyMapToMemStore(HRegion.java:3621)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lambda$writeMiniBatchOperationsToMemStore$0(HRegion.java:3311)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation$$Lambda$475/1305519523.visit(Unknown Source)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations(HRegion.java:3240)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3303)
at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3821)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4248)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4179)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4109)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1059)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:991)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:954)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2833)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
{code}
> MVCC's writeEntry didn't complete and make MVCC stuck
> -----------------------------------------------------
>
> Key: HBASE-22608
> URL: https://issues.apache.org/jira/browse/HBASE-22608
> Project: HBase
> Issue Type: Bug
> Reporter: Guanghao Zhang
> Priority: Critical
>
> {code:java}
> 2019-06-20,05:03:44,917 ERROR [RpcServer.default.RWQ.Fifo.write.handler=61,queue=1,port=22600] org.apache.hadoop.hbase.regionserver.HRegion: Asked to modify this region's (xxxxxx,,1560481375170.10b01c12d58ce75c9aaf1ac15cc2a7f3.) memStoreSizing to a negative value which is incorrect. Current memStoreSizing=-1686222, delta=1489930
> java.lang.Exception
> at org.apache.hadoop.hbase.regionserver.HRegion.checkNegativeMemStoreDataSize(HRegion.java:1317)
> at org.apache.hadoop.hbase.regionserver.HRegion.incMemStoreSize(HRegion.java:1295)
> at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3316)
> at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3821)
> at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4248)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4179)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4109)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1059)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:991)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:954)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2833)
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> {code}
> See [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3728]
> {code:java}
> @Override
> public WriteEntry writeMiniBatchOperationsToMemStore(
> final MiniBatchOperationInProgress<Mutation> miniBatchOp, @Nullable WriteEntry writeEntry)
> throws IOException {
> if (writeEntry == null) {
> writeEntry = region.mvcc.begin();
> }
> super.writeMiniBatchOperationsToMemStore(miniBatchOp, writeEntry.getWriteNumber());
> return writeEntry;
> }
> {code}
> super.writeMiniBatchOperationsToMemStore throw a exception and the new writeEntry cannot be complete and make the MVCC stuck.
>
> And we meet this problem when enable in-memory compaction. But that should be another issue and need to dig more.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)