You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2009/03/10 07:00:50 UTC
[jira] Created: (HADOOP-5448) JobHistory should use JobTracker's
configuration for paths created/read on hadoop.job.history.location
JobHistory should use JobTracker's configuration for paths created/read on hadoop.job.history.location
------------------------------------------------------------------------------------------------------
Key: HADOOP-5448
URL: https://issues.apache.org/jira/browse/HADOOP-5448
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Devaraj Das
The JobHistory methods use the user's jobconf to create/read history file paths on both hadoop.job.history.location and hadoop.job.history.user.location. Even the RecoveryManager uses the user's jobconf. This should be fixed to use the JobTracker's conf for the history files on hadoop.job.history.location.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5448) JobHistory should use JobTracker's
configuration for paths created/read on hadoop.job.history.location
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680385#action_12680385 ]
Amar Kamat commented on HADOOP-5448:
------------------------------------
Job's configuration is also required for _user-name_, _job-name_ and _user-log-location_. Does it make sense to fix (finalize) *hadoop.job.history.location* during JobHistory initialization (JobHistory.init())?
> JobHistory should use JobTracker's configuration for paths created/read on hadoop.job.history.location
> ------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5448
> URL: https://issues.apache.org/jira/browse/HADOOP-5448
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Devaraj Das
>
> The JobHistory methods use the user's jobconf to create/read history file paths on both hadoop.job.history.location and hadoop.job.history.user.location. Even the RecoveryManager uses the user's jobconf. This should be fixed to use the JobTracker's conf for the history files on hadoop.job.history.location.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5448) JobHistory should use JobTracker's
configuration for paths created/read on hadoop.job.history.location
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680386#action_12680386 ]
Amar Kamat commented on HADOOP-5448:
------------------------------------
Also the history filesystem should be finalized.
> JobHistory should use JobTracker's configuration for paths created/read on hadoop.job.history.location
> ------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5448
> URL: https://issues.apache.org/jira/browse/HADOOP-5448
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Devaraj Das
>
> The JobHistory methods use the user's jobconf to create/read history file paths on both hadoop.job.history.location and hadoop.job.history.user.location. Even the RecoveryManager uses the user's jobconf. This should be fixed to use the JobTracker's conf for the history files on hadoop.job.history.location.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5448) JobHistory should use JobTracker's
configuration for paths created/read on hadoop.job.history.location
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5448:
-------------------------------
Attachment: HADOOP-5448-v1.2.patch
Attaching a patch that should fix the issue. This patch determines the FileSystem for {{hadoop.job.history.location}} during init and uses this filesystem everywhere instead of computing it everytime. Result of test-patch
{code}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Test case is bit tricky as JobClient itself uses the job's fs info to upload job files. Ideally the testcase should hack the jobclient code to upload files with faulty *fs.default.name*.
Note that some part of HADOOP-5328 is used here.
> JobHistory should use JobTracker's configuration for paths created/read on hadoop.job.history.location
> ------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5448
> URL: https://issues.apache.org/jira/browse/HADOOP-5448
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Devaraj Das
> Attachments: HADOOP-5448-v1.2.patch
>
>
> The JobHistory methods use the user's jobconf to create/read history file paths on both hadoop.job.history.location and hadoop.job.history.user.location. Even the RecoveryManager uses the user's jobconf. This should be fixed to use the JobTracker's conf for the history files on hadoop.job.history.location.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5448) JobHistory should use JobTracker's
configuration for paths created/read on hadoop.job.history.location
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat resolved HADOOP-5448.
--------------------------------
Resolution: Duplicate
Incorporated in HADOOP-5737.
> JobHistory should use JobTracker's configuration for paths created/read on hadoop.job.history.location
> ------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5448
> URL: https://issues.apache.org/jira/browse/HADOOP-5448
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Devaraj Das
> Attachments: HADOOP-5448-v1.2.patch
>
>
> The JobHistory methods use the user's jobconf to create/read history file paths on both hadoop.job.history.location and hadoop.job.history.user.location. Even the RecoveryManager uses the user's jobconf. This should be fixed to use the JobTracker's conf for the history files on hadoop.job.history.location.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5448) JobHistory should use JobTracker's
configuration for paths created/read on hadoop.job.history.location
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683856#action_12683856 ]
Amar Kamat commented on HADOOP-5448:
------------------------------------
Looks like there are other places where something similar is done.
# JobInProgress.initTasks() does
## {{FileSystem fs = sysDir.getFileSystem(conf);}}
## {{ FileSystem fileSys = tmpDir.getFileSystem(conf);}}
# JobInProgress.garbageCollect() does
## FileSystem fs = tempDir.getFileSystem(conf);
## FileSystem fileSys = tmpDir.getFileSystem(conf);
Searching for 'getFileSystem' in src/org/apache/hadoop/mapreduce reveals places there this is flawed.
> JobHistory should use JobTracker's configuration for paths created/read on hadoop.job.history.location
> ------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5448
> URL: https://issues.apache.org/jira/browse/HADOOP-5448
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Devaraj Das
> Attachments: HADOOP-5448-v1.2.patch
>
>
> The JobHistory methods use the user's jobconf to create/read history file paths on both hadoop.job.history.location and hadoop.job.history.user.location. Even the RecoveryManager uses the user's jobconf. This should be fixed to use the JobTracker's conf for the history files on hadoop.job.history.location.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.