You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amar Kamat (JIRA)" <ji...@apache.org> on 2008/10/08 10:29:44 UTC
[jira] Created: (HADOOP-4372) Improve the way the job history files
are managed during job recovery
Improve the way the job history files are managed during job recovery
---------------------------------------------------------------------
Key: HADOOP-4372
URL: https://issues.apache.org/jira/browse/HADOOP-4372
Project: Hadoop Core
Issue Type: Improvement
Components: mapred
Reporter: Amar Kamat
Assignee: Amar Kamat
Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4372) Improve the way the job history
files are managed during job recovery
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705588#action_12705588 ]
Amar Kamat commented on HADOOP-4372:
------------------------------------
Changes to {{src/mapred/mapred-default.conf}} are not required. Plz ignore that.
> Improve the way the job history files are managed during job recovery
> ---------------------------------------------------------------------
>
> Key: HADOOP-4372
> URL: https://issues.apache.org/jira/browse/HADOOP-4372
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-4372-v1.4.patch, HADOOP-4372-v1.patch, HADOOP-4372-v3.0.patch
>
>
> Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4372) Improve the way the job history files
are managed during job recovery
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4372:
-------------------------------
Attachment: HADOOP-4372-v1.4.patch
Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Ant tests passed on my box.
> Improve the way the job history files are managed during job recovery
> ---------------------------------------------------------------------
>
> Key: HADOOP-4372
> URL: https://issues.apache.org/jira/browse/HADOOP-4372
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-4372-v1.4.patch, HADOOP-4372-v1.patch
>
>
> Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4372) Improve the way the job history files
are managed during job recovery
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4372:
-------------------------------
Attachment: HADOOP-4372-v3.0.patch
Attaching a patch that simply uses the boolean passed whether the job is old or new. Testing in progress.
> Improve the way the job history files are managed during job recovery
> ---------------------------------------------------------------------
>
> Key: HADOOP-4372
> URL: https://issues.apache.org/jira/browse/HADOOP-4372
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-4372-v1.4.patch, HADOOP-4372-v1.patch, HADOOP-4372-v3.0.patch
>
>
> Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4372) Improve the way the job history
files are managed during job recovery
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707251#action_12707251 ]
Ravi Gummadi commented on HADOOP-4372:
--------------------------------------
Patch looks good.
+1
> Improve the way the job history files are managed during job recovery
> ---------------------------------------------------------------------
>
> Key: HADOOP-4372
> URL: https://issues.apache.org/jira/browse/HADOOP-4372
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-4372-v1.4.patch, HADOOP-4372-v1.patch, HADOOP-4372-v3.0.patch, HADOOP-4372-v3.1.patch
>
>
> Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4372) Improve the way the job history
files are managed during job recovery
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707498#action_12707498 ]
Hudson commented on HADOOP-4372:
--------------------------------
Integrated in Hadoop-trunk #830 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/830/])
. Improves the way history filenames are obtained and manipulated. Contributed by Amar Kamat.
> Improve the way the job history files are managed during job recovery
> ---------------------------------------------------------------------
>
> Key: HADOOP-4372
> URL: https://issues.apache.org/jira/browse/HADOOP-4372
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-4372-v1.4.patch, HADOOP-4372-v1.patch, HADOOP-4372-v3.0.patch, HADOOP-4372-v3.1.patch
>
>
> Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-4372) Improve the way the job history
files are managed during job recovery
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das resolved HADOOP-4372.
---------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
I just committed this. Thanks, Amar!
> Improve the way the job history files are managed during job recovery
> ---------------------------------------------------------------------
>
> Key: HADOOP-4372
> URL: https://issues.apache.org/jira/browse/HADOOP-4372
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-4372-v1.4.patch, HADOOP-4372-v1.patch, HADOOP-4372-v3.0.patch, HADOOP-4372-v3.1.patch
>
>
> Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4372) Improve the way the job history
files are managed during job recovery
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718606#action_12718606 ]
Hudson commented on HADOOP-4372:
--------------------------------
Integrated in Hadoop-trunk #863 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/863/])
> Improve the way the job history files are managed during job recovery
> ---------------------------------------------------------------------
>
> Key: HADOOP-4372
> URL: https://issues.apache.org/jira/browse/HADOOP-4372
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-4372-v1.4.patch, HADOOP-4372-v1.patch, HADOOP-4372-v3.0.patch, HADOOP-4372-v3.1.patch
>
>
> Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4372) Improve the way the job history
files are managed during job recovery
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711523#action_12711523 ]
Devaraj Das commented on HADOOP-4372:
-------------------------------------
I committed this to 0.20 branch as well. There have been problems like job submission taking a long time when the number of files in the history folder are too large. This patch introduced an API that would fasten the process of obtaining a history file for a new job (earlier it used to do a scan of the history folder which is costly when the number of files are large).
> Improve the way the job history files are managed during job recovery
> ---------------------------------------------------------------------
>
> Key: HADOOP-4372
> URL: https://issues.apache.org/jira/browse/HADOOP-4372
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-4372-v1.4.patch, HADOOP-4372-v1.patch, HADOOP-4372-v3.0.patch, HADOOP-4372-v3.1.patch
>
>
> Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4372) Improve the way the job history files
are managed during job recovery
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4372:
-------------------------------
Attachment: HADOOP-4372-v1.patch
Attaching a patch the implements the basic idea. This patch uses a new filename on recovery. Updated the testcase accordingly.
> Improve the way the job history files are managed during job recovery
> ---------------------------------------------------------------------
>
> Key: HADOOP-4372
> URL: https://issues.apache.org/jira/browse/HADOOP-4372
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-4372-v1.patch
>
>
> Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4372) Improve the way the job history
files are managed during job recovery
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700130#action_12700130 ]
Devaraj Das commented on HADOOP-4372:
-------------------------------------
Could we instead pass a boolean to logSubmitted depending on the restart-count value (0 == new job, and the boolean is true), and then create the recovery file inside the jobHistory if the boolean is false.
The problem with the current patch is that the filename of the history file changes upon restart..
> Improve the way the job history files are managed during job recovery
> ---------------------------------------------------------------------
>
> Key: HADOOP-4372
> URL: https://issues.apache.org/jira/browse/HADOOP-4372
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-4372-v1.4.patch, HADOOP-4372-v1.patch
>
>
> Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4372) Improve the way the job history files
are managed during job recovery
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-4372:
-------------------------------
Attachment: HADOOP-4372-v3.1.patch
Attaching a patch that optimizes jobhistory for new jobs. Result of test-patch
{code}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Ant test passed on my box
> Improve the way the job history files are managed during job recovery
> ---------------------------------------------------------------------
>
> Key: HADOOP-4372
> URL: https://issues.apache.org/jira/browse/HADOOP-4372
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Attachments: HADOOP-4372-v1.4.patch, HADOOP-4372-v1.patch, HADOOP-4372-v3.0.patch, HADOOP-4372-v3.1.patch
>
>
> Today we use the _.recover_ technique to handle the job history files when the jobtracker restarts. The comment [here|https://issues.apache.org/jira/browse/HADOOP-3245?focusedCommentId=12629080#action_12629080] proposes a better way to handle the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.