You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2010/06/22 04:40:55 UTC
[jira] Created: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
IPC leaks call parameters when exceptions thrown
------------------------------------------------
Key: HADOOP-6833
URL: https://issues.apache.org/jira/browse/HADOOP-6833
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.20.2, 0.21.0, 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Attachments: hadoop-6833.txt
HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-6833:
--------------------------------
Status: Patch Available (was: Open)
> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
> Key: HADOOP-6833
> URL: https://issues.apache.org/jira/browse/HADOOP-6833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903200#action_12903200 ]
Hudson commented on HADOOP-6833:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #366 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/366/])
HADOOP-6833. IPC leaks call parameters when exceptions thrown. Contributed by Todd Lipcon.
> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
> Key: HADOOP-6833
> URL: https://issues.apache.org/jira/browse/HADOOP-6833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.20.3, 0.21.1, 0.22.0
>
> Attachments: hadoop-6833-20.patch, hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-6833:
--------------------------------
Attachment: hadoop-6833.txt
> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
> Key: HADOOP-6833
> URL: https://issues.apache.org/jira/browse/HADOOP-6833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902729#action_12902729 ]
Eli Collins commented on HADOOP-6833:
-------------------------------------
Hey Todd,
Looks like the test added in this patch is racy, it failed for me after I looped it for a couple minutes. When System.gc returns it just means the gc has made a best effort to run, not that it actually has, so in the cases where the gc hasn't run the test will fail because myArg was not reaped.
The change itself looks correct. Since this is a one-liner and memory leaks like this are hard to test I'd be OK with a new patch that just fixes the bug as long as it passes the existing RPC tests. Any other suggestions on how to add test coverage of course are welcome, doesn't seem worth eg adding a getter that returns the number of calls.
Thanks,
Eli
> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
> Key: HADOOP-6833
> URL: https://issues.apache.org/jira/browse/HADOOP-6833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-6833:
--------------------------------
Attachment: hadoop-6833-20.patch
Attaching patch for branch-0.20.
> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
> Key: HADOOP-6833
> URL: https://issues.apache.org/jira/browse/HADOOP-6833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-6833-20.patch, hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903195#action_12903195 ]
Eli Collins commented on HADOOP-6833:
-------------------------------------
+1
Here are the test-patch results. Not sure why it -1 javadoc, ant javadoc|javadoc-dev are clean.
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no new tests are needed for this patch.
[exec] Also please list what manual steps were performed to verify this patch.
[exec]
[exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}
> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
> Key: HADOOP-6833
> URL: https://issues.apache.org/jira/browse/HADOOP-6833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881035#action_12881035 ]
Hadoop QA commented on HADOOP-6833:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12447656/hadoop-6833.txt
against trunk revision 956710.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
-1 javac. The applied patch generated 1017 javac compiler warnings (more than the trunk's current 1013 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/590/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/590/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/590/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/590/console
This message is automatically generated.
> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
> Key: HADOOP-6833
> URL: https://issues.apache.org/jira/browse/HADOOP-6833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903370#action_12903370 ]
Hudson commented on HADOOP-6833:
--------------------------------
Integrated in Hadoop-Common-trunk #435 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/435/])
HADOOP-6833. IPC leaks call parameters when exceptions thrown. Contributed by Todd Lipcon.
> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
> Key: HADOOP-6833
> URL: https://issues.apache.org/jira/browse/HADOOP-6833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.20.3, 0.21.1, 0.22.0
>
> Attachments: hadoop-6833-20.patch, hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881038#action_12881038 ]
Todd Lipcon commented on HADOOP-6833:
-------------------------------------
I think the "new warnings" are because I used UTF8 in the test, and UTF8 is deprecated. I used it to follow the pattern elsewhere in the test. Let me know if I should change the test case to use something like Text or LongWritable instead.
> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
> Key: HADOOP-6833
> URL: https://issues.apache.org/jira/browse/HADOOP-6833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-6833:
--------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Fix Version/s: 0.20.3
0.21.1
0.22.0
Resolution: Fixed
I committed this to 20, 21 and trunk. Thanks Todd!
> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
> Key: HADOOP-6833
> URL: https://issues.apache.org/jira/browse/HADOOP-6833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: 0.20.3, 0.21.1, 0.22.0
>
> Attachments: hadoop-6833-20.patch, hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6833) IPC leaks call parameters when
exceptions thrown
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-6833:
--------------------------------
Attachment: hadoop-6833.txt
Here's the same patch minus the unit test.
As for manual/cluster testing, we've been running with this fix in HBase's copy of RPC for a couple months. We used to eventually run out of RAM, heap analysis showed this was the problem. This fix got rid of the issue.
> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
> Key: HADOOP-6833
> URL: https://issues.apache.org/jira/browse/HADOOP-6833
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.