[jira] [Created] (HBASE-29390) Too many logs in AsyncBatchRpcRetryingCaller when hitting RegionTooBusyException

["Duo Zhang (Jira)" <ji...@apache.org>]

Duo Zhang created HBASE-29390:
---------------------------------

             Summary: Too many logs in AsyncBatchRpcRetryingCaller when hitting RegionTooBusyException
                 Key: HBASE-29390
                 URL: https://issues.apache.org/jira/browse/HBASE-29390
             Project: HBase
          Issue Type: Improvement
            Reporter: Duo Zhang


When checking the iTBLL result, I saw bunch of logs in the link generator logs like this

{noformat}
2025-06-10 06:52:30,800 WARN [RPCClient-NioEventLoopGroup-1-8] org.apache.hadoop.hbase.client.AsyncBatchRpcRetryingCaller: Process batch for ['IntegrationTestBigLinkedList,<"\x0F\x91\xF3\x07\x83\x07\xDBI\xF8\xB5\xCA\xD6\x19\xB8,1749399341229.6a9162
4d74729cdd03e61ee7500e6951.'] in IntegrationTestBigLinkedList from data01,16020,1749509062898 failed, tries=6
org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 M, regionName=6a91624d74729cdd03e61ee7500e6951, server=data01,16020,1749509062898
        at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:5379)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4840)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4766)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1021)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:939)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:904)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2877)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45448)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:457)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
        at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
        at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)

        at jdk.internal.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
        at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1725)
        at org.apache.hadoop.hbase.shaded.protobuf.ResponseConverter.getResults(ResponseConverter.java:172)
        at org.apache.hadoop.hbase.client.AsyncBatchRpcRetryingCaller.lambda$sendToServer$11(AsyncBatchRpcRetryingCaller.java:409)
        at org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:56)
        at org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:47)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:400)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:430)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
        at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:117)
        at org.apache.hadoop.hbase.ipc.Call.setResponse(Call.java:149)
        at org.apache.hadoop.hbase.ipc.RpcConnection.finishCall(RpcConnection.java:396)
        at org.apache.hadoop.hbase.ipc.RpcConnection.readResponse(RpcConnection.java:461)
        at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:125)
        at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:140)
        at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
        at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
        at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
        at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
{noformat}

The same output repeated many many times, which makes it really difficult to find useful logs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)