You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ramya R (JIRA)" <ji...@apache.org> on 2010/07/30 06:41:16 UTC
[jira] Created: (MAPREDUCE-1986) History files fail to move to DONE
folder when hadoop.job.history.location is configured to a HDFS path
History files fail to move to DONE folder when hadoop.job.history.location is configured to a HDFS path
-------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-1986
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1986
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.20.2
Reporter: Ramya R
When hadoop.job.history.location is configured to a HDFS path and when the job retires, the history files are never moved to
mapred.job.tracker.history.completed.location and they remain forever in hadoop.job.history.location . The jobhistory is never viewable thereafter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1986) History files fail to move to
DONE folder when hadoop.job.history.location is configured to a HDFS path
Posted by "Ramya R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893929#action_12893929 ]
Ramya R commented on MAPREDUCE-1986:
------------------------------------
I have the following setup on my cluster:
<property>
<name>hadoop.job.history.location</name>
<value>/mapred/history1</value>
<description> If job tracker is static the history files are stored in this single well known place. If No
value is set here, by default,
it is in the local file system at ${hadoop.log.dir}/history.
</description></property>
<property>
<name>mapred.job.tracker.history.completed.location</name>
<value>/mapred/history2/done</value>
<description> The completed job history files are stored at this single well known location. If nothing is
specified, the files are stored at
${hadoop.job.history.location}/done.
</description>
</property>
i.e. both hadoop.job.history.location and mapred.job.tracker.history.completed.location are configured to HDFS path.
The jobtracker successfully comes up without any error. However when the job completes and it retires, I get the following exception:
{noformat}
ERROR org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE canonical subfolder.
java.io.FileNotFoundException: File /mapred/history1/job_201007290838_0001_1280392750589_rramya_Sleep+job does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1118)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1094)
at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1084)
at org.apache.hadoop.mapred.JobHistory$JobHistoryFilesManager$1.run(JobHistory.java:300)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
However, /mapred/history1 is created and the history files are very much present on HDFS.
Thus the history files remain forever in hadoop.job.history.location and are never moved to mapred.job.tracker.history.completed.location. The jobhistory is never viewable thereafter.
> History files fail to move to DONE folder when hadoop.job.history.location is configured to a HDFS path
> -------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1986
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1986
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Ramya R
>
> When hadoop.job.history.location is configured to a HDFS path and when the job retires, the history files are never moved to
> mapred.job.tracker.history.completed.location and they remain forever in hadoop.job.history.location . The jobhistory is never viewable thereafter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.