You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Hong Tang (JIRA)" <ji...@apache.org> on 2009/09/18 00:29:57 UTC
[jira] Created: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
JobHistory.initDone() should retain the try ... catch in the body
-----------------------------------------------------------------
Key: MAPREDUCE-1000
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Hong Tang
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759515#action_12759515 ]
Hudson commented on MAPREDUCE-1000:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #65 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/65/])
. Handle corrupt history files in JobHistory.initDone(). Contributed by Jothi Padmanabhan.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758653#action_12758653 ]
Hong Tang commented on MAPREDUCE-1000:
--------------------------------------
Patch looks good. One minor nit, pls use junit4 in the newly introduced unit test.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Attachments: mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1000:
-----------------------------------------
Status: Open (was: Patch Available)
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1000:
-----------------------------------------
Status: Patch Available (was: Open)
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759026#action_12759026 ]
Jothi Padmanabhan commented on MAPREDUCE-1000:
----------------------------------------------
Test failures are related to HADOOP-6281. WIll try Hudson once that is committed.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1000:
-----------------------------------------
Status: Patch Available (was: Open)
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Attachments: mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756946#action_12756946 ]
Hong Tang commented on MAPREDUCE-1000:
--------------------------------------
That is a fair argument. It seems that we could ignore some IOE in moveOldFiles() and ensure the end results of this call is that all files are gone (either moved, or removed).
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759432#action_12759432 ]
Jothi Padmanabhan commented on MAPREDUCE-1000:
----------------------------------------------
Failing test, TestCopyFiles is a known issue.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759406#action_12759406 ]
Hadoop QA commented on MAPREDUCE-1000:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch
against trunk revision 818674.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/console
This message is automatically generated.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759057#action_12759057 ]
Hadoop QA commented on MAPREDUCE-1000:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch
against trunk revision 818355.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/console
This message is automatically generated.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758688#action_12758688 ]
Hadoop QA commented on MAPREDUCE-1000:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12420354/mapred-1000.patch
against trunk revision 817740.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/console
This message is automatically generated.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756956#action_12756956 ]
Jothi Padmanabhan commented on MAPREDUCE-1000:
----------------------------------------------
bq. It seems that we could ignore some IOE in moveOldFiles() and ensure the end results of this call is that all files are gone (either moved, or removed).
This looks like a good idea
bq. Since checksum exceptions should be reasonably common for partial history files, would it make sense to use RawLocalFileSystem as the src fs in moveOldFiles?
Actually, the job history log location is governed by the conf setting "hadoop.job.history.location" and this could possibly be on a FS other than LocalFS. So, I do not think we could use RawLocalFileSystem.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758659#action_12758659 ]
Sharad Agarwal commented on MAPREDUCE-1000:
-------------------------------------------
There is already a test case in TestJobHistory.testDoneFolderOnHDFS() which tests the moveToDone() code path on start up. Instead of adding a full new MiniMRCluster based testcase I think we can enhance a bit to test this. Also it would be good to assert that corrupt files got deleted from the current log directory and didn't move to done folder.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Attachments: mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756952#action_12756952 ]
Chris Douglas commented on MAPREDUCE-1000:
------------------------------------------
Since checksum exceptions should be reasonably common for partial history files, would it make sense to use RawLocalFileSystem as the src fs in moveOldFiles?
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1000:
-----------------------------------------
Attachment: mapred-1000-v2.patch
Patch that does the delete only on checksum exception.
Also, moved the test case inside TestJobHistory so that job history test cases are consolidated in one place.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan reassigned MAPREDUCE-1000:
--------------------------------------------
Assignee: Jothi Padmanabhan
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758813#action_12758813 ]
Hadoop QA commented on MAPREDUCE-1000:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch
against trunk revision 817740.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/console
This message is automatically generated.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1000:
-----------------------------------------
Status: Open (was: Patch Available)
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1000:
-----------------------------------------
Status: Patch Available (was: Open)
Trying Hudson again
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1000:
-----------------------------------------
Status: Patch Available (was: Open)
The findbugs warning is spurious. The same patch gave +1 for findbugs in the previous run. TestCopyFiles failure is a known failure. Running by Hudson again, just in case
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756826#action_12756826 ]
Hong Tang commented on MAPREDUCE-1000:
--------------------------------------
MAPREDUCE-157 changed JobHistory.initDone() and removed the try...catch clause of the body. The try...catch body is necessary because otherwise, if an IOE is thrown during the execution, JT would be aborted. I observed it when testing MAPREDUCE-728.
Symptom:
{noformat}
org.apache.hadoop.fs.ChecksumException: Checksum error: file:/Users/htang/Documents/Work/workspace/hadoop-mapreduce/build/hadoop-mapred-0.21.0-dev/logs/history/job_200904211745_0010_geek5 at 523264
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:221)
at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:238)
at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:190)
at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:72)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:45)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:97)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:220)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:143)
at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:55)
at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1203)
at org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:338)
at org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveOldFiles(JobHistory.java:372)
at org.apache.hadoop.mapreduce.jobhistory.JobHistory.initDone(JobHistory.java:145)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:3900)
at org.apache.hadoop.mapred.SimulatorJobTracker.<init>(SimulatorJobTracker.java:80)
{noformat}
The previous run of the JT was killed, which leaves the job history file mismatching with CRC checksum.
The selected patch segment that shows the removal of the try...catch clause:
Before MAPREDUCE-157
{noformat}
- static boolean initDone(JobConf conf, FileSystem fs){
- try {
- //if completed job history location is set, use that
- String doneLocation = conf.
- get("mapred.job.tracker.history.completed.location");
- if (doneLocation != null) {
- DONE = fs.makeQualified(new Path(doneLocation));
- DONEDIR_FS = fs;
- } else {
- DONE = new Path(LOG_DIR, "done");
- DONEDIR_FS = LOGDIR_FS;
- }
-
- //If not already present create the done folder with appropriate
- //permission
- if (!DONEDIR_FS.exists(DONE)) {
- LOG.info("Creating DONE folder at "+ DONE);
- if (! DONEDIR_FS.mkdirs(DONE,
- new FsPermission(HISTORY_DIR_PERMISSION))) {
- throw new IOException("Mkdirs failed to create " + DONE.toString());
- }
- }
-
- fileManager.start();
- //move the log files remaining from last run to the DONE folder
- //suffix the file name based on Jobtracker identifier so that history
- //files with same job id don't get over written in case of recovery.
- FileStatus[] files = LOGDIR_FS.listStatus(new Path(LOG_DIR));
- String jtIdentifier = fileManager.jobTracker.getTrackerIdentifier();
- String fileSuffix = "." + jtIdentifier + OLD_SUFFIX;
- for (FileStatus fileStatus : files) {
- Path fromPath = fileStatus.getPath();
- if (fromPath.equals(DONE)) { //DONE can be a subfolder of log dir
- continue;
- }
- LOG.info("Moving log file from last run: " + fromPath);
- Path toPath = new Path(DONE, fromPath.getName() + fileSuffix);
- fileManager.moveToDoneNow(fromPath, toPath);
- }
- } catch(IOException e) {
- LOG.error("Failed to initialize JobHistory log file", e);
- disableHistory = true;
- }
- return !(disableHistory);
- }
{noformat}
After MAPREDUCE-157
{noformat}
+ /** Initialize the done directory and start the history cleaner thread */
+ public void initDone(JobConf conf, FileSystem fs) throws IOException {
+ //if completed job history location is set, use that
+ String doneLocation =
+ conf.get("mapred.job.tracker.history.completed.location");
+ if (doneLocation != null) {
+ done = fs.makeQualified(new Path(doneLocation));
+ doneDirFs = fs;
+ } else {
+ done = logDirFs.makeQualified(new Path(logDir, "done"));
+ doneDirFs = logDirFs;
+ }
+
+ //If not already present create the done folder with appropriate
+ //permission
+ if (!doneDirFs.exists(done)) {
+ LOG.info("Creating DONE folder at "+ done);
+ if (! doneDirFs.mkdirs(done,
+ new FsPermission(HISTORY_DIR_PERMISSION))) {
+ throw new IOException("Mkdirs failed to create " + done.toString());
+ }
+ }
+ LOG.info("Inited the done directory to " + done.toString());
+
+ moveOldFiles();
+ startFileMoverThreads();
+
+ // Start the History Cleaner Thread
+ long maxAgeOfHistoryFiles = conf.getLong(
+ "mapreduce.cluster.jobhistory.maxage", DEFAULT_HISTORY_MAX_AGE);
+ historyCleanerThread = new HistoryCleaner(maxAgeOfHistoryFiles);
+ historyCleanerThread.start();
+ }
{noformat}
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756943#action_12756943 ]
Jothi Padmanabhan commented on MAPREDUCE-1000:
----------------------------------------------
bq,. IOE is thrown during the execution, JT would be aborted
Actually we intentionally did that -- catch Exceptions during History Initialization and exit JT. Otherwise, these exceptions could go totally unnoticed and we might end up with no history file at all. Infact, this was one of Sharad's review comments [here|https://issues.apache.org/jira/browse/MAPREDUCE-157?focusedCommentId=12753542&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12753542]
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758684#action_12758684 ]
Jothi Padmanabhan commented on MAPREDUCE-1000:
----------------------------------------------
Had an offline discussion with Sharad. We think we should actually delete only in the case of Checksum Exception. For other exceptions, we should just log and continue. For example, if the done directory does not have permissions for write, we might get an exception while trying to move and we do not want to delete in that case. Thoughts?
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759015#action_12759015 ]
Hadoop QA commented on MAPREDUCE-1000:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch
against trunk revision 818355.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/console
This message is automatically generated.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated MAPREDUCE-1000:
--------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
+1
I just committed this to 0.21 branch and trunk. Thanks Jothi.
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000-v2.patch, mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1000:
-----------------------------------------
Status: Open (was: Patch Available)
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Attachments: mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1000:
-----------------------------------------
Component/s: jobtracker
Affects Version/s: 0.21.0
Fix Version/s: 0.21.0
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Fix For: 0.21.0
>
> Attachments: mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1000) JobHistory.initDone() should
retain the try ... catch in the body
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1000:
-----------------------------------------
Attachment: mapred-1000.patch
Patch that tries to delete the history on exception while trying to move them to done folder.
Also attached a test case that demonstrates the fix -- it fails with trunk and passes with the patch
> JobHistory.initDone() should retain the try ... catch in the body
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
> Assignee: Jothi Padmanabhan
> Attachments: mapred-1000.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.