You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Ted Yu (JIRA)" <ji...@apache.org> on 2012/11/02 18:23:12 UTC
[jira] [Created] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Ted Yu created HBASE-7086:
-----------------------------
Summary: Enhance ResourceChecker to log stack trace for potentially hanging threads
Key: HBASE-7086
URL: https://issues.apache.org/jira/browse/HBASE-7086
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
{code}
2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
{code}
We should enhance the log to include stack trace of the potentially hanging thread(s)
This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-7086:
---------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490129#comment-13490129 ]
Ted Yu commented on HBASE-7086:
-------------------------------
Integrated to trunk.
Thanks for the review, Lars and Stack.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-7086:
--------------------------
Fix Version/s: 0.96.0
0.94.3
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu reassigned HBASE-7086:
-----------------------------
Assignee: Ted Yu
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489587#comment-13489587 ]
stack commented on HBASE-7086:
------------------------------
Pity the logs are cut off. I trust you. +1 on commit to 0.94. Trunk patch would be sweet.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Attachments: 7086.94
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489948#comment-13489948 ]
Lars Hofhansl commented on HBASE-7086:
--------------------------------------
The 0.94 part of this is good, right? Let's move the 0.96 part to a new jira, so I can close this for the next RC. (Unless the trunk part gets resolve soon)
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489581#comment-13489581 ]
Ted Yu commented on HBASE-7086:
-------------------------------
Here is sample output from TestHFileCleaner-output in 0.94 branch:
{code}
2012-11-02 10:18:59,231 INFO [main] hbase.ResourceChecker(157): before master.cleaner.TestHFileCleaner#testTTLCleaner: 43 threads, 145 file descriptors 0 connections,
2012-11-02 10:18:59,297 DEBUG [main] cleaner.TimeToLiveHFileCleaner(68): Life:117, ttl:100, current:1351876739296, from: 1351876739179
2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(180): after master.cleaner.TestHFileCleaner#testTTLCleaner: potentially hanging thread
2012-11-02 10:18:59,300 INFO [main] hbase.ResourceChecker(186): java.lang.Thread.sleep(Native Method)
2012-11-02 10:18:59,300 INFO [main] hbase.ResourceChecker(186): org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)
2012-11-02 10:18:59,301 INFO [main] hbase.ResourceChecker(186): java.lang.Thread.run(Thread.java:680)
2012-11-02 10:18:59,305 INFO [main] hbase.ResourceChecker(157): before master.cleaner.TestHFileCleaner#testHFileCleaning: 44 threads, 145 file descriptors 0 connections,
{code}
Current goal is to figure out why TestHFileCleaner#testTTLCleaner hangs in 0.94
Trunk patch would be quite different from 0.94 patch.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Attachments: 7086.94
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-7086:
--------------------------
Status: Patch Available (was: Open)
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-7086:
--------------------------
Status: Patch Available (was: Open)
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489938#comment-13489938 ]
Hadoop QA commented on HBASE-7086:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12551938/7086-trunk-v3.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 85 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3218//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3218//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3218//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3218//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3218//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3218//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3218//console
This message is automatically generated.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-7086:
--------------------------
Attachment: 7086.94
Patch for 0.94 branch.
Trunk has different class structure.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Attachments: 7086.94
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489613#comment-13489613 ]
Ted Yu commented on HBASE-7086:
-------------------------------
Integrated to 0.94 branch.
Working on trunk patch now.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489771#comment-13489771 ]
Hudson commented on HBASE-7086:
-------------------------------
Integrated in HBase-0.94 #567 (See [https://builds.apache.org/job/HBase-0.94/567/])
HBASE-7086 Enhance ResourceChecker to log stack trace for potentially hanging threads (Revision 1405081)
Result = FAILURE
tedyu :
Files :
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ResourceCheckerJUnitRule.java
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489821#comment-13489821 ]
Ted Yu commented on HBASE-7086:
-------------------------------
Integrated addendum for 0.94 to 0.94 branch.
Thanks for the review, Lars.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489605#comment-13489605 ]
Lars Hofhansl commented on HBASE-7086:
--------------------------------------
Patch looks good. +1
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-7086:
--------------------------
Attachment: 7086-trunk.txt
Patch for trunk.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-7086:
--------------------------
Attachment: 7086-94.addendum
Addendum for 0.94 which allows better grouping of stack trace by thread.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490378#comment-13490378 ]
Hudson commented on HBASE-7086:
-------------------------------
Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/])
HBASE-7086 Enhance ResourceChecker to log stack trace for potentially hanging threads, addendum (Revision 1405207)
HBASE-7086 Enhance ResourceChecker to log stack trace for potentially hanging threads (Revision 1405081)
Result = FAILURE
tedyu :
Files :
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java
tedyu :
Files :
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ResourceCheckerJUnitRule.java
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489994#comment-13489994 ]
Ted Yu commented on HBASE-7086:
-------------------------------
I will create new Jira if 0.96 portion is not checked in by Monday.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490118#comment-13490118 ]
Lars Hofhansl commented on HBASE-7086:
--------------------------------------
trunk v3 looks to me. +1
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489571#comment-13489571 ]
stack commented on HBASE-7086:
------------------------------
Does it work? Does it retain the above resource checker log line and list the difference in threads after this logging? How we make this work in trunk? I like stuff like this that gives more info when testing.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Attachments: 7086.94
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489665#comment-13489665 ]
stack commented on HBASE-7086:
------------------------------
Trunk patch looks good (see what hadoopqa says)
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490230#comment-13490230 ]
Hudson commented on HBASE-7086:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #248 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/248/])
HBASE-7086 Enhance ResourceChecker to log stack trace for potentially hanging threads (Revision 1405443)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java
* /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/ResourceCheckerJUnitListener.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ServerResourceCheckerJUnitListener.java
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-7086:
--------------------------
Status: Open (was: Patch Available)
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-7086:
--------------------------
Attachment: testHFileCleaner.out
The test output recorded earlier this morning.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-7086:
--------------------------
Attachment: 7086-trunk-v3.txt
Trunk patch v3 illustrates how the stack trace is logged.
Here is sample from test output:
{code}
2012-11-02 15:46:35,429 INFO [main] hbase.ResourceChecker(147): before: master.cleaner.TestHFileCleaner#testTTLCleaner Thread=43, OpenFileDescriptor=145, MaxFileDescriptor=10240, ConnectionCount=0
2012-11-02 15:46:35,671 DEBUG [main] cleaner.TimeToLiveHFileCleaner(68): Life:106, ttl:100, current:1351896395669, from: 1351896395563
2012-11-02 15:46:35,673 INFO [main] hbase.ResourceChecker(171): after: master.cleaner.TestHFileCleaner#testTTLCleaner Thread=44 (was 43)
Potentially hanging thread: LeaseChecker
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1485)
java.lang.Thread.run(Thread.java:680)
- Thread LEAK? -, OpenFileDescriptor=145 (was 145), MaxFileDescriptor=10240 (was 10240), ConnectionCount=0 (was 0)
2012-11-02 15:46:35,674 INFO [main] hbase.ResourceChecker(147): before: master.cleaner.TestHFileCleaner#testHFileCleaning Thread=44, OpenFileDescriptor=145, MaxFileDescriptor=10240, ConnectionCount=0
{code}
I am open to the naming of the new method in ResourceChecker.ResourceAnalyzer
Please provide your comments.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489747#comment-13489747 ]
Lars Hofhansl commented on HBASE-7086:
--------------------------------------
+1 on addendum and trunk patch
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7086) Enhance ResourceChecker to log stack
trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-7086:
--------------------------
Attachment: 7086-trunk-v2.txt
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489931#comment-13489931 ]
Hudson commented on HBASE-7086:
-------------------------------
Integrated in HBase-0.94 #568 (See [https://builds.apache.org/job/HBase-0.94/568/])
HBASE-7086 Enhance ResourceChecker to log stack trace for potentially hanging threads, addendum (Revision 1405207)
Result = FAILURE
tedyu :
Files :
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, 7086-trunk-v3.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7086) Enhance ResourceChecker to log
stack trace for potentially hanging threads
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489774#comment-13489774 ]
Ted Yu commented on HBASE-7086:
-------------------------------
Looks like I cannot access https://builds.apache.org/job/PreCommit-HBASE-Build/ at the moment.
Recent Jenkins builds returned strange exception as well.
My first trunk patch didn't work. There would be a lot of extraneous log such as the following:
{code}
NumberFormatException: 2012-11-02 14:23:11,497 DEBUG [pool-1-thread-1] backup.HFileArchiver(338): No existing file in archive for:/home/hduser/trunk/hbase-server/target/test-data/9e6c26d7-45f9-406f-87eb-a733231256ac/testWithMinVersions/.archive/testWithMinVersions/07e89fc98af6b9300cd5c8e4c19fa8d9/colfamily31/a3644ad438fd40f885b29959730c1fde, free to archive original file.
NumberFormatException: 2012-11-02 14:23:11,497 DEBUG [pool-1-thread-1] backup.HFileArchiver(345): Finished archiving file from: class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:/home/hduser/trunk/hbase-server/target/test-data/9e6c26d7-45f9-406f-87eb-a733231256ac/testWithMinVersions/testWithMinVersions/07e89fc98af6b9300cd5c8e4c19fa8d9/colfamily31/a3644ad438fd40f885b29959730c1fde, to: /home/hduser/trunk/hbase-server/target/test-data/9e6c26d7-45f9-406f-87eb-a733231256ac/testWithMinVersions/.archive/testWithMinVersions/07e89fc98af6b9300cd5c8e4c19fa8d9/colfamily31/a3644ad438fd40f885b29959730c1fde
{code}
In trunk patch v2, I pass Log object to ResourceCheckerJUnitListener.
It seems that the actual log couldn't be written when there is thread leak:
{code}
2012-11-02 14:31:58,084 INFO [main] hbase.ResourceChecker(162): after: io.hfile.TestScannerSelectionUsingTTL#testScannerSelection[3] Thread=11 (was 10) - Thread LEAK? -, OpenFileDescriptor=104 (was 102) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), ConnectionCount=0 (was 0)
2012-11-02 14:31:58,085 INFO [main] hbase.ResourceChecker(144): before: io.hfile.TestScannerSelectionUsingTTL#testScannerSelection[4] Thread=11, OpenFileDescriptor=104, MaxFileDescriptor=10240, ConnectionCount=0
{code}
My next step is to add method to ResourceChecker.ResourceAnalyzer which returns array of String so that ResourceChecker can log them.
But I want to get N Keywal's input first.
> Enhance ResourceChecker to log stack trace for potentially hanging threads
> --------------------------------------------------------------------------
>
> Key: HBASE-7086
> URL: https://issues.apache.org/jira/browse/HBASE-7086
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7086.94, 7086-94.addendum, 7086-trunk.txt, 7086-trunk-v2.txt, testHFileCleaner.out
>
>
> Currently ResourceChecker logs a line similar to the following if it detects potential thread leak:
> {code}
> 2012-11-02 10:18:59,299 INFO [main] hbase.ResourceChecker(157): after master.cleaner.TestHFileCleaner#testTTLCleaner: 44 threads (was 43), 145 file descriptors (was 145). 0 connections, -thread leak?-
> {code}
> We should enhance the log to include stack trace of the potentially hanging thread(s)
> This work was motivated when I investigated test failure in HBASE-6796
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira