[jira] [Commented] (HBASE-5141) Memory leak in MonitoredRPCHandlerImpl

["Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181697#comment-13181697 ] 

Todd Lipcon commented on HBASE-5141:
------------------------------------

Part of me thinks this is a configuration error -- a conservative admin should always assume that the memory usage of IPC buffers = numHandlers*maxPayload, since you could always have the threads all concurrently handling large calls. So memory should be allocated for this. The fact that the memory is kept around until the next call makes it more likely you'd hit this, but it's still a potential problem regardless.

I think the proper fix for this problem is to keep an atomic counter for the amount of memory used by IPC handlers, and gate the read calls off the wire based on a memory budget.
                
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
>                 Key: HBASE-5141
>                 URL: https://issues.apache.org/jira/browse/HBASE-5141
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.92.0
>            Reporter: Jean-Daniel Cryans
>            Priority: Blocker
>             Fix For: 0.92.0, 0.94.0
>
>         Attachments: Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira