You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Guanghao Zhang (JIRA)" <ji...@apache.org> on 2019/06/20 01:24:00 UTC
[jira] [Updated] (HBASE-22608) MVCC's writeEntry didn't complete
and make MVCC stuck
[ https://issues.apache.org/jira/browse/HBASE-22608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guanghao Zhang updated HBASE-22608:
-----------------------------------
Description:
{code:java}
2019-06-20,05:03:44,917 ERROR [RpcServer.default.RWQ.Fifo.write.handler=61,queue=1,port=22600] org.apache.hadoop.hbase.regionserver.HRegion: Asked to modify this region's (xxxxxx,,1560481375170.10b01c12d58ce75c9aaf1ac15cc2a7f3.) memStoreSizing to a negative value which is incorrect. Current memStoreSizing=-1686222, delta=1489930
java.lang.Exception
at org.apache.hadoop.hbase.regionserver.HRegion.checkNegativeMemStoreDataSize(HRegion.java:1317)
at org.apache.hadoop.hbase.regionserver.HRegion.incMemStoreSize(HRegion.java:1295)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3316)
at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3821)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4248)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4179)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4109)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1059)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:991)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:954)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2833)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
{code}
See [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3728]
{code:java}
@Override
public WriteEntry writeMiniBatchOperationsToMemStore(
final MiniBatchOperationInProgress<Mutation> miniBatchOp, @Nullable WriteEntry writeEntry)
throws IOException {
if (writeEntry == null) {
writeEntry = region.mvcc.begin();
}
super.writeMiniBatchOperationsToMemStore(miniBatchOp, writeEntry.getWriteNumber());
return writeEntry;
}
{code}
super.writeMiniBatchOperationsToMemStore throw a exception and the new writeEntry cannot be complete and make the MVCC stuck.
And we meet this problem when enable in-memory compaction. But that should be another issue and need to dig more.
was:
{code:java}
2019-06-20,05:03:44,917 ERROR [RpcServer.default.RWQ.Fifo.write.handler=61,queue=1,port=22600] org.apache.hadoop.hbase.regionserver.HRegion: Asked to modify this region's (xxxxxx,,1560481375170.10b01c12d58ce75c9aaf1ac15cc2a7f3.) memStoreSizing to a negative value which is incorrect. Current memStoreSizing=-1686222, delta=1489930
java.lang.Exception
at org.apache.hadoop.hbase.regionserver.HRegion.checkNegativeMemStoreDataSize(HRegion.java:1317)
at org.apache.hadoop.hbase.regionserver.HRegion.incMemStoreSize(HRegion.java:1295)
at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3316)
at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3821)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4248)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4179)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4109)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1059)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:991)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:954)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2833)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
{code}
See https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3728
{code:java}
@Override
public WriteEntry writeMiniBatchOperationsToMemStore(
final MiniBatchOperationInProgress<Mutation> miniBatchOp, @Nullable WriteEntry writeEntry)
throws IOException {
if (writeEntry == null) {
writeEntry = region.mvcc.begin();
}
super.writeMiniBatchOperationsToMemStore(miniBatchOp, writeEntry.getWriteNumber());
return writeEntry;
}
{code}
super.writeMiniBatchOperationsToMemStore throw a exception and the new writeEntry cannot be complete and make the MVCC stuck.
> MVCC's writeEntry didn't complete and make MVCC stuck
> -----------------------------------------------------
>
> Key: HBASE-22608
> URL: https://issues.apache.org/jira/browse/HBASE-22608
> Project: HBase
> Issue Type: Bug
> Reporter: Guanghao Zhang
> Priority: Major
>
> {code:java}
> 2019-06-20,05:03:44,917 ERROR [RpcServer.default.RWQ.Fifo.write.handler=61,queue=1,port=22600] org.apache.hadoop.hbase.regionserver.HRegion: Asked to modify this region's (xxxxxx,,1560481375170.10b01c12d58ce75c9aaf1ac15cc2a7f3.) memStoreSizing to a negative value which is incorrect. Current memStoreSizing=-1686222, delta=1489930
> java.lang.Exception
> at org.apache.hadoop.hbase.regionserver.HRegion.checkNegativeMemStoreDataSize(HRegion.java:1317)
> at org.apache.hadoop.hbase.regionserver.HRegion.incMemStoreSize(HRegion.java:1295)
> at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3316)
> at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3821)
> at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4248)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4179)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4109)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1059)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:991)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:954)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2833)
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> {code}
> See [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3728]
> {code:java}
> @Override
> public WriteEntry writeMiniBatchOperationsToMemStore(
> final MiniBatchOperationInProgress<Mutation> miniBatchOp, @Nullable WriteEntry writeEntry)
> throws IOException {
> if (writeEntry == null) {
> writeEntry = region.mvcc.begin();
> }
> super.writeMiniBatchOperationsToMemStore(miniBatchOp, writeEntry.getWriteNumber());
> return writeEntry;
> }
> {code}
> super.writeMiniBatchOperationsToMemStore throw a exception and the new writeEntry cannot be complete and make the MVCC stuck.
>
> And we meet this problem when enable in-memory compaction. But that should be another issue and need to dig more.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)