[jira] [Commented] (HBASE-30092) MonitoredRPCHandlerImpl.generateCallInfoMap() can OOM the RegionServer heartbeat path when an in-flight RPC request protobuf is large

["mazhengxuan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-30092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097514#comment-18097514 ] 

mazhengxuan commented on HBASE-30092:
-------------------------------------

I would like to investigate this issue. I will check the MonitoredRPCHandlerImpl.generateCallInfoMap path and try to prepare a bounded fix to avoid excessive memory usage. If nobody is working on it, could I try to fix it?

> MonitoredRPCHandlerImpl.generateCallInfoMap() can OOM the RegionServer heartbeat path when an in-flight RPC request protobuf is large
> -------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-30092
>                 URL: https://issues.apache.org/jira/browse/HBASE-30092
>             Project: HBase
>          Issue Type: Bug
>          Components: monitoring, regionserver
>    Affects Versions: 2.6.5, 2.5.14
>            Reporter: Nick Dimiduk
>            Priority: Major
>
> A burst of large {{MultiRequest}} / {{ScanRequest}} RPCs on a RegionServer with a tight heap crashed the JVM with {{OutOfMemoryError}} from the periodic RegionServer-to-HMaster heartbeat path. The offending allocation is inside {{MonitoredRPCHandlerImpl.generateCallInfoMap()}}, which calls {{AbstractMessage.toString()}} on the live in-flight RPC request protobuf. {{TextFormat$Printer}} recursively appends to a {{StringBuilder}} that has to grow past available heap.
> {noformat}
> java.lang.OutOfMemoryError
>   at java.util.Arrays.copyOf
>   at java.lang.AbstractStringBuilder.ensureCapacityInternal
>   at java.lang.StringBuilder.append (CharSequence)
>   at org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$TextGenerator.eol
>   at org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField
>   at org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$Printer.printField
>   at org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$Printer.printMessage
>   ... [recursive printMessage / printField for nested protobuf fields] ...
>   at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage.toString  (AbstractMessage.java:91)
>   at org.apache.hadoop.hbase.monitoring.MonitoredRPCHandlerImpl.generateCallInfoMap   (MonitoredRPCHandlerImpl.java:253)
>   at org.apache.hadoop.hbase.monitoring.MonitoredRPCHandlerImpl.clone                (MonitoredRPCHandlerImpl.java:57)
>   at org.apache.hadoop.hbase.monitoring.TaskMonitor.processTasks                     (TaskMonitor.java:229)
>   at org.apache.hadoop.hbase.monitoring.TaskMonitor.getTasks                         (TaskMonitor.java:183)
>   at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad              (HRegionServer.java:1566)
>   at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport
>   at org.apache.hadoop.hbase.regionserver.HRegionServer.run
> {noformat}
> {{tryRegionServerReport}} runs on the main RegionServer thread at {{hbase.regionserver.msginterval}} (default 3 s). An OOM there terminates the JVM under {{-XX:OnOutOfMemoryError}}, or aborts the heartbeat otherwise. On a heap sized close to the working set, a brief write slowdown that causes handlers to back up is enough to grow per-handler scanner retention and push {{generateCallInfoMap()}} over the edge. Auto-replacement by an external pool manager doesn't break the loop — each replacement RS inherits the same client traffic and crashes again within 15–60 seconds of registering.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)