You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2008/07/21 09:03:31 UTC
[jira] Issue Comment Edited: (HADOOP-3245) Provide ability to
persist running jobs (extend HADOOP-1876)
[ https://issues.apache.org/jira/browse/HADOOP-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615158#action_12615158 ]
dhruba edited comment on HADOOP-3245 at 7/21/08 12:02 AM:
--------------------------------------------------------------------
+1 to Owen's comment.
A pressing need of our cluster is to not interrupt running jobs if the entire cluster has to be restarted. This means that job states have to be persisted in the form of a transaction log. This requirement is all the more beneficial to sites that have long-running job trackers (instead of HOD).
However, isn't it better to be able to store state in HDFS? It is true that HDFS stores its transaction log in local files, but with the current focus on improving HDFS read/write latencies, HDFS itself is considering whether to store one copy of the transaction log in HDFS blocks (instead of NFS). In fact, if the JobTracker stores information in a org.apache.hadoop.fs.FileSystem, then a typical customer install could plug in various forms of storage to support the JobTracker transaction log.
was (Author: dhruba):
+1 to Owen's comment.
A pressing need of our cluster is to not interrupt running jobs if the jobtracker has to be restarted. This means that job states have to be persisted in the form of a transaction log. This requirement is all the more beneficial to sites that have long-running job trackers (instead of HOD).
However, isn't it better to be able to store state in HDFS? It is true that HDFS stores its transaction log in local files, but with the current focus on improving HDFS read/write latencies, HDFS itself is considering whether to store one copy of the transaction log in HDFS blocks (instead of NFS). In fact, if the JobTracker stores information in a org.apache.hadoop.fs.FileSystem, then a typical customer install could plug in various forms of storage to support the JobTracker transaction log.
> Provide ability to persist running jobs (extend HADOOP-1876)
> ------------------------------------------------------------
>
> Key: HADOOP-3245
> URL: https://issues.apache.org/jira/browse/HADOOP-3245
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Amar Kamat
> Attachments: HADOOP-3245-v2.5.patch, HADOOP-3245-v2.6.5.patch, HADOOP-3245-v2.6.9.patch, HADOOP-3245-v4.1.patch
>
>
> This could probably extend the work done in HADOOP-1876. This feature can be applied for things like jobs being able to survive jobtracker restarts.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.