You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Kevin Wilfong (Created) (JIRA)" <ji...@apache.org> on 2012/03/09 04:33:59 UTC
[jira] [Created] (HIVE-2858) Cache remote map reduce job stack
traces for additional logging
Cache remote map reduce job stack traces for additional logging
---------------------------------------------------------------
Key: HIVE-2858
URL: https://issues.apache.org/jira/browse/HIVE-2858
Project: Hive
Issue Type: Improvement
Components: Logging
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2858) Cache remote map reduce job stack
traces for additional logging
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2858:
------------------------------
Attachment: HIVE-2858.D2223.1.patch
kevinwilfong requested code review of "HIVE-2858 [jira] Cache remote map reduce job stack traces for additional logging".
Reviewers: JIRA
https://issues.apache.org/jira/browse/HIVE-2858
Modified the JobDebugger and TaskLogProcessor to parse out stack traces from failed task logs. These are stored in a mapping from job ID to stack traces in the SessionState.
Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D2223
AFFECTED FILES
conf/hive-default.xml.template
build-common.xml
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out
ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out
ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java
ql/src/test/queries/clientnegative/mapreduce_stack_trace.q
ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java
ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java
ql/src/java/org/apache/hadoop/hive/ql/Driver.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/4875/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Cache remote map reduce job stack traces for additional logging
> ---------------------------------------------------------------
>
> Key: HIVE-2858
> URL: https://issues.apache.org/jira/browse/HIVE-2858
> Project: Hive
> Issue Type: Improvement
> Components: Logging
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2858.D2223.1.patch
>
>
> Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2858) Cache remote map reduce job stack
traces for additional logging
Posted by "Ashutosh Chauhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HIVE-2858:
-----------------------------------
Fix Version/s: 0.9.0
> Cache remote map reduce job stack traces for additional logging
> ---------------------------------------------------------------
>
> Key: HIVE-2858
> URL: https://issues.apache.org/jira/browse/HIVE-2858
> Project: Hive
> Issue Type: Improvement
> Components: Logging
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Fix For: 0.9.0
>
> Attachments: HIVE-2858.D2223.1.patch, HIVE-2858.D2223.2.patch
>
>
> Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2858) Cache remote map reduce job stack
traces for additional logging
Posted by "Namit Jain (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-2858:
-----------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed. Thanks Kevin
> Cache remote map reduce job stack traces for additional logging
> ---------------------------------------------------------------
>
> Key: HIVE-2858
> URL: https://issues.apache.org/jira/browse/HIVE-2858
> Project: Hive
> Issue Type: Improvement
> Components: Logging
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2858.D2223.1.patch, HIVE-2858.D2223.2.patch
>
>
> Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2858) Cache remote map reduce job stack
traces for additional logging
Posted by "Namit Jain (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-2858:
-----------------------------
Status: Open (was: Patch Available)
> Cache remote map reduce job stack traces for additional logging
> ---------------------------------------------------------------
>
> Key: HIVE-2858
> URL: https://issues.apache.org/jira/browse/HIVE-2858
> Project: Hive
> Issue Type: Improvement
> Components: Logging
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2858.D2223.1.patch
>
>
> Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2858) Cache remote map reduce job stack
traces for additional logging
Posted by "Kevin Wilfong (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-2858:
--------------------------------
Status: Patch Available (was: Open)
> Cache remote map reduce job stack traces for additional logging
> ---------------------------------------------------------------
>
> Key: HIVE-2858
> URL: https://issues.apache.org/jira/browse/HIVE-2858
> Project: Hive
> Issue Type: Improvement
> Components: Logging
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2858.D2223.1.patch, HIVE-2858.D2223.2.patch
>
>
> Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2858) Cache remote map reduce job stack
traces for additional logging
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249218#comment-13249218 ]
Hudson commented on HIVE-2858:
------------------------------
Integrated in Hive-trunk-h0.21 #1360 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1360/])
HIVE-2858 Cache remote map reduce job stack traces for additional
logging (Kevin Wilfong via namit) (Revision 1310583)
Result = ABORTED
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310583
Files :
* /hive/trunk/build-common.xml
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml.template
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java
* /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace.q
* /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q
* /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out
* /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out
> Cache remote map reduce job stack traces for additional logging
> ---------------------------------------------------------------
>
> Key: HIVE-2858
> URL: https://issues.apache.org/jira/browse/HIVE-2858
> Project: Hive
> Issue Type: Improvement
> Components: Logging
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2858.D2223.1.patch, HIVE-2858.D2223.2.patch
>
>
> Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2858) Cache remote map reduce job stack
traces for additional logging
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13243998#comment-13243998 ]
Phabricator commented on HIVE-2858:
-----------------------------------
njain has commented on the revision "HIVE-2858 [jira] Cache remote map reduce job stack traces for additional logging".
can you refresh and re-apply the patch ?
It is not applying cleanly for me.
REVISION DETAIL
https://reviews.facebook.net/D2223
> Cache remote map reduce job stack traces for additional logging
> ---------------------------------------------------------------
>
> Key: HIVE-2858
> URL: https://issues.apache.org/jira/browse/HIVE-2858
> Project: Hive
> Issue Type: Improvement
> Components: Logging
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2858.D2223.1.patch
>
>
> Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2858) Cache remote map reduce job stack
traces for additional logging
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2858:
------------------------------
Attachment: HIVE-2858.D2223.2.patch
kevinwilfong updated the revision "HIVE-2858 [jira] Cache remote map reduce job stack traces for additional logging".
Reviewers: JIRA, njain
Updated and fixed the conflicts.
REVISION DETAIL
https://reviews.facebook.net/D2223
AFFECTED FILES
conf/hive-default.xml.template
build-common.xml
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out
ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out
ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java
ql/src/test/queries/clientnegative/mapreduce_stack_trace.q
ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java
ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java
ql/src/java/org/apache/hadoop/hive/ql/Driver.java
> Cache remote map reduce job stack traces for additional logging
> ---------------------------------------------------------------
>
> Key: HIVE-2858
> URL: https://issues.apache.org/jira/browse/HIVE-2858
> Project: Hive
> Issue Type: Improvement
> Components: Logging
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2858.D2223.1.patch, HIVE-2858.D2223.2.patch
>
>
> Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2858) Cache remote map reduce job stack
traces for additional logging
Posted by "Kevin Wilfong (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-2858:
--------------------------------
Status: Patch Available (was: Open)
> Cache remote map reduce job stack traces for additional logging
> ---------------------------------------------------------------
>
> Key: HIVE-2858
> URL: https://issues.apache.org/jira/browse/HIVE-2858
> Project: Hive
> Issue Type: Improvement
> Components: Logging
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2858.D2223.1.patch
>
>
> Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira