You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2010/01/26 19:04:34 UTC
[jira] Created: (MAPREDUCE-1413) Improve logging of progress/errors
in the job and task trackers
Improve logging of progress/errors in the job and task trackers
---------------------------------------------------------------
Key: MAPREDUCE-1413
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: tasktracker
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
I have code that improves the logging of the trackers as they start stop and fail, through
# More logging of events
# including exception strings and stacks when things go wrong
People's whose JTs and TTs aren't behaving may appreciate this
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1413) Improve logging of progress/errors
in the job and task trackers
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1413:
-------------------------------------
Status: Open (was: Patch Available)
* All the error log messages added include the exception twice, but once is sufficient.
* The javadoc on {{toString}} refers to the state of the JobTracker, not the TaskTracker.
> Improve logging of progress/errors in the job and task trackers
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1413
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1413.patch
>
>
> I have code that improves the logging of the trackers as they start stop and fail, through
> # More logging of events
> # including exception strings and stacks when things go wrong
> People's whose JTs and TTs aren't behaving may appreciate this
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1413) Improve logging of progress/errors
in the job and task trackers
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated MAPREDUCE-1413:
--------------------------------------
Affects Version/s: 0.22.0
Fix Version/s: 0.22.0
> Improve logging of progress/errors in the job and task trackers
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1413
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1413.patch
>
>
> I have code that improves the logging of the trackers as they start stop and fail, through
> # More logging of events
> # including exception strings and stacks when things go wrong
> People's whose JTs and TTs aren't behaving may appreciate this
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1413) Improve logging of
progress/errors in the job and task trackers
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833756#action_12833756 ]
Steve Loughran commented on MAPREDUCE-1413:
-------------------------------------------
The log calls aren't logging the _whole_ exception twice
# The log message includes the toString() value of the exception, normally the value of getMessage(), though that can be changed. This message doesn't include the stack trace
# The second parameter passes in the raw exception chain with full stack trace. It is up to the log engine what it does with that; printing out the stack trace in text is one option, but I know of others (discard, save the fault as an XML-parseable). This is why log(text,Throwable) is the recommended way to log faults in all the main java log frameworks.
# Sometimes I've seen loggers not log the toString() value of the exception passed in as param #2, which is less than ideal if you are trying to track down a problem and all you have are the logs.
# Therefore, I do think what I've done is the best tactic to producing logs that are human and machine readable.
# Will check out the javadoc error and submit a patch that corrects that.
> Improve logging of progress/errors in the job and task trackers
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1413
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1413.patch
>
>
> I have code that improves the logging of the trackers as they start stop and fail, through
> # More logging of events
> # including exception strings and stacks when things go wrong
> People's whose JTs and TTs aren't behaving may appreciate this
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1413) Improve logging of progress/errors
in the job and task trackers
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated MAPREDUCE-1413:
--------------------------------------
Status: Patch Available (was: In Progress)
This patch doesn't include tests for logged stuff, though one for the toString() operator is possible.
> Improve logging of progress/errors in the job and task trackers
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1413
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1413.patch
>
>
> I have code that improves the logging of the trackers as they start stop and fail, through
> # More logging of events
> # including exception strings and stacks when things go wrong
> People's whose JTs and TTs aren't behaving may appreciate this
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1413) Improve logging of progress/errors
in the job and task trackers
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated MAPREDUCE-1413:
--------------------------------------
Attachment: MAPREDUCE-1413.patch
> Improve logging of progress/errors in the job and task trackers
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1413
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1413.patch
>
>
> I have code that improves the logging of the trackers as they start stop and fail, through
> # More logging of events
> # including exception strings and stacks when things go wrong
> People's whose JTs and TTs aren't behaving may appreciate this
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (MAPREDUCE-1413) Improve logging of
progress/errors in the job and task trackers
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on MAPREDUCE-1413 started by Steve Loughran.
> Improve logging of progress/errors in the job and task trackers
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1413
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
>
> I have code that improves the logging of the trackers as they start stop and fail, through
> # More logging of events
> # including exception strings and stacks when things go wrong
> People's whose JTs and TTs aren't behaving may appreciate this
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1413) Improve logging of
progress/errors in the job and task trackers
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834105#action_12834105 ]
Chris Douglas commented on MAPREDUCE-1413:
------------------------------------------
That seems tautological to me: if one's logger is configured to not print the exception, then it is unavailable in the logs. But as a practical accommodation, I see your point. OK.
Instead of defining {{toString()}} for the TaskTracker, could this just print the relevant information in offerService? Or are you using that in the service lifecycle branch somehow?
> Improve logging of progress/errors in the job and task trackers
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1413
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1413.patch
>
>
> I have code that improves the logging of the trackers as they start stop and fail, through
> # More logging of events
> # including exception strings and stacks when things go wrong
> People's whose JTs and TTs aren't behaving may appreciate this
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1413) Improve logging of
progress/errors in the job and task trackers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805270#action_12805270 ]
Hadoop QA commented on MAPREDUCE-1413:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12431454/MAPREDUCE-1413.patch
against trunk revision 902727.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/287/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/287/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/287/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/287/console
This message is automatically generated.
> Improve logging of progress/errors in the job and task trackers
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1413
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1413.patch
>
>
> I have code that improves the logging of the trackers as they start stop and fail, through
> # More logging of events
> # including exception strings and stacks when things go wrong
> People's whose JTs and TTs aren't behaving may appreciate this
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1413) Improve logging of
progress/errors in the job and task trackers
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805852#action_12805852 ]
Steve Loughran commented on MAPREDUCE-1413:
-------------------------------------------
The tests that this patch failed were cluster blacklisting. odd
> Improve logging of progress/errors in the job and task trackers
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1413
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1413.patch
>
>
> I have code that improves the logging of the trackers as they start stop and fail, through
> # More logging of events
> # including exception strings and stacks when things go wrong
> People's whose JTs and TTs aren't behaving may appreciate this
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.