You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Duo Zhang (Jira)" <ji...@apache.org> on 2025/06/12 11:29:00 UTC

[jira] [Commented] (HBASE-29390) Too many logs in AsyncBatchRpcRetryingCaller when hitting RegionTooBusyException

    [ https://issues.apache.org/jira/browse/HBASE-29390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17966040#comment-17966040 ] 

Duo Zhang commented on HBASE-29390:
-----------------------------------

I think the problem is here

https://github.com/apache/hbase/blob/54442b12e7ca583eeabb4d68174416964ac923b1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBatchRpcRetryingCaller.java#L336

For each action, will call the onComplete method, and in the onComplete method

https://github.com/apache/hbase/blob/54442b12e7ca583eeabb4d68174416964ac923b1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBatchRpcRetryingCaller.java#L310

We will call logException for every action.

The error in description is RegionTooBusyException, so all the actions should fail with RegionTooBusyException, so after tries reaches the threshold, for every action we will log the exception, which will flood the log output.

Let me think how to fix it properly...

> Too many logs in AsyncBatchRpcRetryingCaller when hitting RegionTooBusyException
> --------------------------------------------------------------------------------
>
>                 Key: HBASE-29390
>                 URL: https://issues.apache.org/jira/browse/HBASE-29390
>             Project: HBase
>          Issue Type: Improvement
>          Components: asyncclient, Client
>            Reporter: Duo Zhang
>            Assignee: Duo Zhang
>            Priority: Major
>
> When checking the iTBLL result, I saw bunch of logs in the link generator logs like this
> {noformat}
> 2025-06-10 06:52:30,800 WARN [RPCClient-NioEventLoopGroup-1-8] org.apache.hadoop.hbase.client.AsyncBatchRpcRetryingCaller: Process batch for ['IntegrationTestBigLinkedList,<"\x0F\x91\xF3\x07\x83\x07\xDBI\xF8\xB5\xCA\xD6\x19\xB8,1749399341229.6a9162
> 4d74729cdd03e61ee7500e6951.'] in IntegrationTestBigLinkedList from data01,16020,1749509062898 failed, tries=6
> org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 M, regionName=6a91624d74729cdd03e61ee7500e6951, server=data01,16020,1749509062898
>         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:5379)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4840)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4766)
>         at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1021)
>         at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:939)
>         at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:904)
>         at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2877)
>         at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45448)
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:457)
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
>         at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
>         at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
>         at jdk.internal.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
>         at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
>         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
>         at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1725)
>         at org.apache.hadoop.hbase.shaded.protobuf.ResponseConverter.getResults(ResponseConverter.java:172)
>         at org.apache.hadoop.hbase.client.AsyncBatchRpcRetryingCaller.lambda$sendToServer$11(AsyncBatchRpcRetryingCaller.java:409)
>         at org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:56)
>         at org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:47)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:400)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:430)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
>         at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:117)
>         at org.apache.hadoop.hbase.ipc.Call.setResponse(Call.java:149)
>         at org.apache.hadoop.hbase.ipc.RpcConnection.finishCall(RpcConnection.java:396)
>         at org.apache.hadoop.hbase.ipc.RpcConnection.readResponse(RpcConnection.java:461)
>         at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:125)
>         at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:140)
>         at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
>         at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>         at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>         at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>         at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>         at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>         at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>         at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>         at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
>         at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
>         at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>         at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> {noformat}
> The same output repeated many many times, which makes it really difficult to find useful logs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)