[jira] Commented: (HADOOP-171) need standard API to set dfs replication = high

["Doug Cutting (JIRA)" <ji...@apache.org>]

    [ http://issues.apache.org/jira/browse/HADOOP-171?page=comments#action_12376604 ] 

Doug Cutting commented on HADOOP-171:
-------------------------------------

So my concrete question is: how should I change JobClient?  It currently uses fs.copyFromLocalFile() to copy the job.jar and fs.create() to write the job.xml.  Both of these files should be replicated highly.  This code should not have to explicitly query the FileSystem to ask how many nodes it has, etc.

If my Short.MAX_VALUE suggestion were implemented, then these calls might become something like:

fs.create("job.xml", Short.MAX_VALUE);

and

fs.copyFromLocalFile(localJobJar, remoteJobJar, Short.MAX_VALUE);

Do someone have another proposal for how we should achieve this?

> need standard API to set dfs replication = high
> -----------------------------------------------
>
>          Key: HADOOP-171
>          URL: http://issues.apache.org/jira/browse/HADOOP-171
>      Project: Hadoop
>         Type: New Feature

>   Components: dfs
>     Versions: 0.2
>     Reporter: Doug Cutting
>     Assignee: Konstantin Shvachko

>
> There should be a standard way to indicate that files should be highly replicated, appropriate for files that all nodes will read.  This should be settable both on file creation and for already-existing files.  Perhaps specifying a particular replication value, like Short.MAX_VALUE, or zero, can be used to signal this.  The level should not be constant, but should be relative to the cluster size and network topography.  If more nodes are added or if nodes are deleted, the actual replication count should increase or decrease.
> Initially, all that is needed is an API to specify this.  It could initially be implemented with a constant (e.g., 10) or with something related to the number of datanodes (sqrt?), and needn't auto-adjust as the cluster size changes  That is only  the long-term goal.
> When JobClient copies job files (job.xml & job.jar) into the job's filesystem, it should specify this replication level.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira