You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (Created) (JIRA)" <ji...@apache.org> on 2012/01/07 00:08:42 UTC
[jira] [Created] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Memory leak in MonitoredRPCHandlerImpl
--------------------------------------
Key: HBASE-5141
URL: https://issues.apache.org/jira/browse/HBASE-5141
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Priority: Blocker
Fix For: 0.92.0, 0.94.0
I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
Will attach a screen shot of jprofiler's analysis in a moment.
This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182127#comment-13182127 ]
Mikhail Bautin commented on HBASE-5141:
---------------------------------------
FYI: The build is broken in trunk because of this patch.
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182136#comment-13182136 ]
Mikhail Bautin commented on HBASE-5141:
---------------------------------------
Correction: use this svn command:
svn diff http://svn.apache.org/repos/asf/hbase/trunk -r1228739
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183970#comment-13183970 ]
Hudson commented on HBASE-5141:
-------------------------------
Integrated in HBase-TRUNK #2617 (See [https://builds.apache.org/job/HBase-TRUNK/2617/])
HBASE-5141 Memory leak in MonitoredRPCHandlerImpl -- REDO
HBASE-5141 Memory leak in MonitoredRPCHandlerImpl -- REVERT. OVER-COMMITTED. REVERTING ALL SO CAN REDO COMMIT
HBASE-5141 Memory leak in MonitoredRPCHandlerImpl
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181705#comment-13181705 ]
Todd Lipcon commented on HBASE-5141:
------------------------------------
Yep, sorry, the memory budgeting would have to be before the data is read into the queue in the IPC server.
I agree that nulling out the rpc packet will fix the issue as reported here. Just saying that the issue doesn't seem likely to affect most use cases where RPCs are of mixed size and where people have already budgeted their heap to fit a bunch of calls.
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-5141:
--------------------------------------
Attachment: Screen Shot 2012-01-06 at 3.03.09 PM.png
This screen shot shows how MonitoredRPCHandlerImpl are all using 6% of the heap because they are holding on the packets.
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182070#comment-13182070 ]
Hadoop QA commented on HBASE-5141:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12509797/HBASE-5141-v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated -151 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 79 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/696//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/696//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/696//console
This message is automatically generated.
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182120#comment-13182120 ]
stack commented on HBASE-5141:
------------------------------
Let me commit....
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181730#comment-13181730 ]
Jean-Daniel Cryans commented on HBASE-5141:
-------------------------------------------
After much testing, it's actually params not packet that has all that data.
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182202#comment-13182202 ]
Hudson commented on HBASE-5141:
-------------------------------
Integrated in HBase-TRUNK-security #67 (See [https://builds.apache.org/job/HBase-TRUNK-security/67/])
HBASE-5141 Memory leak in MonitoredRPCHandlerImpl -- REDO
HBASE-5141 Memory leak in MonitoredRPCHandlerImpl -- REVERT. OVER-COMMITTED. REVERTING ALL SO CAN REDO COMMIT
HBASE-5141 Memory leak in MonitoredRPCHandlerImpl
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182103#comment-13182103 ]
stack commented on HBASE-5141:
------------------------------
+1 on the patch. Small.
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181701#comment-13181701 ]
Zhihong Yu commented on HBASE-5141:
-----------------------------------
One solution is to release the packet before WritableRpcEngine.call() returns ?
{noformat}
status.setRPCPacket(null);
return retVal;
{noformat}
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181697#comment-13181697 ]
Todd Lipcon commented on HBASE-5141:
------------------------------------
Part of me thinks this is a configuration error -- a conservative admin should always assume that the memory usage of IPC buffers = numHandlers*maxPayload, since you could always have the threads all concurrently handling large calls. So memory should be allocated for this. The fact that the memory is kept around until the next call makes it more likely you'd hit this, but it's still a potential problem regardless.
I think the proper fix for this problem is to keep an atomic counter for the amount of memory used by IPC handlers, and gate the read calls off the wire based on a memory budget.
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5141:
-------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182215#comment-13182215 ]
Hudson commented on HBASE-5141:
-------------------------------
Integrated in HBase-0.92-security #67 (See [https://builds.apache.org/job/HBase-0.92-security/67/])
HBASE-5141 Memory leak in MonitoredRPCHandlerImpl
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182140#comment-13182140 ]
stack commented on HBASE-5141:
------------------------------
Fixed. Sorry about that Mikhail. Thanks for flagging it.
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182135#comment-13182135 ]
Mikhail Bautin commented on HBASE-5141:
---------------------------------------
Actually, the committed patch contains more stuff than the patch attached to the JIRA:
svn diff http://svn.apache.org/repos/asf/hbase/trunk -r1228740
Was the security version of the patch committed into trunk or something?
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-5141:
--------------------------------------
Attachment: HBASE-5141.patch
Testing this patch, I like it better than passing null as the monitor takes care of itself.
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-5141:
--------------------------------------
Assignee: Jean-Daniel Cryans
Status: Patch Available (was: Open)
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182122#comment-13182122 ]
stack commented on HBASE-5141:
------------------------------
Committed 0.92 and trunk.
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-5141:
--------------------------------------
Attachment: HBASE-5141-v2.patch
This second patch survives my little test. What I was missing was that the packet also contains a reference to the params, so I have to clear out both (that was a bit confusing).
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182160#comment-13182160 ]
Hudson commented on HBASE-5141:
-------------------------------
Integrated in HBase-0.92 #233 (See [https://builds.apache.org/job/HBase-0.92/233/])
HBASE-5141 Memory leak in MonitoredRPCHandlerImpl
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182132#comment-13182132 ]
Mikhail Bautin commented on HBASE-5141:
---------------------------------------
I get the error shown at http://pastebin.com/AdAp0M35 when trying to build the following commit:
Author: stack <st...@13f79535-47bb-0310-9956-ffa450edef68>
Date: Sat Jan 7 14:16:11 2012
HBASE-5141 Memory leak in MonitoredRPCHandlerImpl
git-svn-id: http://svn.apache.org/repos/asf/hbase/trunk@1228740
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181702#comment-13181702 ]
Jean-Daniel Cryans commented on HBASE-5141:
-------------------------------------------
bq. I think the proper fix for this problem is to keep an atomic counter for the amount of memory used by IPC handlers, and gate the read calls off the wire based on a memory budget.
There's more to it, it's not just the handlers that have data but their queues too. By default we add 10 for each handler. I believe this is unrelated to this jira's issue.
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in
MonitoredRPCHandlerImpl
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182139#comment-13182139 ]
stack commented on HBASE-5141:
------------------------------
Fixing....
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB in my case), a default heap and default number of handlers, it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values will kill themselves.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira