You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2008/03/06 09:24:58 UTC
[jira] Created: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
[HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
---------------------------------------------------------------------------------
Key: HADOOP-2947
URL: https://issues.apache.org/jira/browse/HADOOP-2947
Project: Hadoop Core
Issue Type: Improvement
Components: contrib/hod
Affects Versions: 0.16.0
Reporter: Hemanth Yamijala
Fix For: 0.17.0
Copied from internal bug details from Koji:
==========================
Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
Please have the stdout of JT/TT redirected to a file.
Adding stderr.
If thread has an uncaught exception, it prints out to stderr and dies.
==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-2947:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Vinod !
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2, HADOOP-2947.3, HADOOP-2947.4
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-2947:
-------------------------------------
Release Note: The stdout and stderr streams of daemons are redirected to files that are created under the hadoop log directory. Users can now send kill 3 signals to the daemons to get stack traces and thread dumps for debugging.
Hadoop Flags: [Reviewed]
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2, HADOOP-2947.3, HADOOP-2947.4
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-2947:
--------------------------------------------
Status: Open (was: Patch Available)
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2, HADOOP-2947.3
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-2947:
--------------------------------------------
Status: Patch Available (was: Open)
Running it through Hudson.
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-2947:
--------------------------------------------
Status: Patch Available (was: Open)
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2, HADOOP-2947.3, HADOOP-2947.4
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala reassigned HADOOP-2947:
----------------------------------------
Assignee: Vinod Kumar Vavilapalli
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.17.0
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-2947:
-------------------------------------
Fix Version/s: (was: 0.17.0)
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Attachments: HADOOP-2947
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2947) [HOD] Hod should redirect stderr
and stdout of Hadoop daemons to assist debugging
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584045#action_12584045 ]
Hemanth Yamijala commented on HADOOP-2947:
------------------------------------------
- The patch assumes current working directory while referring to the helper program. This will not work
- Can we also change the stdout test to use the helper program. I like having the expected output of the test under our complete control. For e.g. say ls is aliased to something in some environment, this test would break again.
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2, HADOOP-2947.3
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-2947:
--------------------------------------------
Attachment: HADOOP-2947.4
Incorporating the suggested changes.
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2, HADOOP-2947.3, HADOOP-2947.4
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-2947:
--------------------------------------------
Status: Open (was: Patch Available)
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2947) [HOD] Hod should redirect stderr
and stdout of Hadoop daemons to assist debugging
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584212#action_12584212 ]
Hadoop QA commented on HADOOP-2947:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379020/HADOOP-2947.4
against trunk revision 643282.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 7 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2112/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2112/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2112/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2112/console
This message is automatically generated.
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2, HADOOP-2947.3, HADOOP-2947.4
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-2947:
--------------------------------------------
Attachment: HADOOP-2947
Attaching a patch. stdout and stderr are now saved along with hadoop.log in the hadoop log directory. Thus, stdout is redirected to $HADOOP_LOG_DIR/<daemon-name>.out and stderr to $HADOOP_LOG_DIR/<daemon-name>.err.
Made changes to simpleCommand to enable starting commands with stderr and/or stdout set. And made related changes in HadoopCommand.
Added testcases. Also tested by giving a sigquit to jobtracker and verifying that the standard output indeed goes to the jobtracker.out file. Didn't get to check stderr.
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-2947:
--------------------------------------------
Attachment: HADOOP-2947.2
Earlier patch kind of does a radical surgery on simpleCommand, though, as Hemanth suggested, this could be done without disturbing much of the current code. Attaching a new patch. This uses the current simpleCommand framework to redirect the stderr and stdout to files. Redirecting these on all daemons to <daemon-name.out> and <daemon-name.err> in the corresponding log directories, which when log-destination-uri is specified, will be archived along with hadoop logs onto dfs.
Tested stdout by doing a sigquit, tested stderr by forcing namenode to fail in intializing(for e.g removed logging jar from hadoop lib dir). Tested to make sure that streams are getting redirected to respective files as desired.
Added test cases also.
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-2947:
--------------------------------------------
Attachment: HADOOP-2947.3
Added a helper program that prints sample text to stderr, and changed the test-case testRedirectedStderr to call this helper program instead.
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2, HADOOP-2947.3
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2947) [HOD] Hod should redirect stderr
and stdout of Hadoop daemons to assist debugging
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583755#action_12583755 ]
Hemanth Yamijala commented on HADOOP-2947:
------------------------------------------
Functionality works well with the patch. Also code looks good.
One minor point is the test cases. The current tests are using ls to check output. The output of these programs could be environment specific, and the tests could fail because of that.
For e.g. I verified on FreeBSD that when ls fails, what comes out on stderr is something like
ls: <filename>: No such file or directory,
whereas on Linux it says:
/bin/ls: <filename>: No such file or directory.
So, I think we should have tests that aren't dependent on platforms like this.
Other than this, +1
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2947) [HOD] Hod should redirect stderr
and stdout of Hadoop daemons to assist debugging
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584195#action_12584195 ]
Hemanth Yamijala commented on HADOOP-2947:
------------------------------------------
+1 for the changes.
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2, HADOOP-2947.3, HADOOP-2947.4
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2947) [HOD] Hod should redirect stderr
and stdout of Hadoop daemons to assist debugging
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581025#action_12581025 ]
Hemanth Yamijala commented on HADOOP-2947:
------------------------------------------
After looking at the changes, we realize there are some issues with the patch. Also, this is requiring changes to a core functionality of HOD. So, moving this out of the 0.17 list.
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Attachments: HADOOP-2947
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2947) [HOD] Hod should redirect stderr
and stdout of Hadoop daemons to assist debugging
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12580263#action_12580263 ]
Hadoop QA commented on HADOOP-2947:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378113/HADOOP-2947
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 4 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1992/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1992/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1992/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1992/console
This message is automatically generated.
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-2947:
-------------------------------------
Priority: Blocker (was: Major)
Fix Version/s: 0.17.0
Marking this as a blocker, after discussions with Mukund and Sameer. We are close to the fix, and can be done in a couple of days, and it seems a useful thing to have.
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2947) [HOD] Hod should redirect stderr and
stdout of Hadoop daemons to assist debugging
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-2947:
--------------------------------------------
Status: Patch Available (was: Open)
> [HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2947
> URL: https://issues.apache.org/jira/browse/HADOOP-2947
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hod
> Affects Versions: 0.16.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: HADOOP-2947, HADOOP-2947.2, HADOOP-2947.3
>
>
> Copied from internal bug details from Koji:
> ==========================
> Sometimes JobTracker/TaskTracker starts consuming 99% cpu and stops responding to 'jstack' call. In those cases,
> usually it still responds to kill -QUIT signal which forces the jvm to dump the stack to stdout.
> Please have the stdout of JT/TT redirected to a file.
> Adding stderr.
> If thread has an uncaught exception, it prints out to stderr and dies.
> ==========================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.