You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Doug Cutting (JIRA)" <ji...@apache.org> on 2006/04/27 01:26:02 UTC
[jira] Created: (HADOOP-171) need standard API to set dfs
replication = high
need standard API to set dfs replication = high
-----------------------------------------------
Key: HADOOP-171
URL: http://issues.apache.org/jira/browse/HADOOP-171
Project: Hadoop
Type: New Feature
Components: dfs
Versions: 0.2
Reporter: Doug Cutting
Assigned to: Konstantin Shvachko
There should be a standard way to indicate that files should be highly replicated, appropriate for files that all nodes will read. This should be settable both on file creation and for already-existing files. Perhaps specifying a particular replication value, like Short.MAX_VALUE, or zero, can be used to signal this. The level should not be constant, but should be relative to the cluster size and network topography. If more nodes are added or if nodes are deleted, the actual replication count should increase or decrease.
Initially, all that is needed is an API to specify this. It could initially be implemented with a constant (e.g., 10) or with something related to the number of datanodes (sqrt?), and needn't auto-adjust as the cluster size changes That is only the long-term goal.
When JobClient copies job files (job.xml & job.jar) into the job's filesystem, it should specify this replication level.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-171) need standard API to set dfs
replication = high
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-171?page=comments#action_12376786 ]
Konstantin Shvachko commented on HADOOP-171:
--------------------------------------------
There are two different issues here, I think
1) asking for a replication hint.
This is what highReplicationHint() is for.
We can provide a parameter (strategy) making it look like
short highReplicationHint( enum strategy )
This one is easy since it just returns a number based on the number of registered data nodes.
But it works for the problem of choosing replication for submitted job files (.jar, .xml).
2) dynamically maintaining replication of a file based on a chosen strategy and cluster size.
This is hard.
Do we really need that now?
Are submitted job files removed after the job is done?
> need standard API to set dfs replication = high
> -----------------------------------------------
>
> Key: HADOOP-171
> URL: http://issues.apache.org/jira/browse/HADOOP-171
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.2
> Reporter: Doug Cutting
> Assignee: Konstantin Shvachko
>
> There should be a standard way to indicate that files should be highly replicated, appropriate for files that all nodes will read. This should be settable both on file creation and for already-existing files. Perhaps specifying a particular replication value, like Short.MAX_VALUE, or zero, can be used to signal this. The level should not be constant, but should be relative to the cluster size and network topography. If more nodes are added or if nodes are deleted, the actual replication count should increase or decrease.
> Initially, all that is needed is an API to specify this. It could initially be implemented with a constant (e.g., 10) or with something related to the number of datanodes (sqrt?), and needn't auto-adjust as the cluster size changes That is only the long-term goal.
> When JobClient copies job files (job.xml & job.jar) into the job's filesystem, it should specify this replication level.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-171) need standard API to set dfs
replication = high
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-171?page=comments#action_12376604 ]
Doug Cutting commented on HADOOP-171:
-------------------------------------
So my concrete question is: how should I change JobClient? It currently uses fs.copyFromLocalFile() to copy the job.jar and fs.create() to write the job.xml. Both of these files should be replicated highly. This code should not have to explicitly query the FileSystem to ask how many nodes it has, etc.
If my Short.MAX_VALUE suggestion were implemented, then these calls might become something like:
fs.create("job.xml", Short.MAX_VALUE);
and
fs.copyFromLocalFile(localJobJar, remoteJobJar, Short.MAX_VALUE);
Do someone have another proposal for how we should achieve this?
> need standard API to set dfs replication = high
> -----------------------------------------------
>
> Key: HADOOP-171
> URL: http://issues.apache.org/jira/browse/HADOOP-171
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.2
> Reporter: Doug Cutting
> Assignee: Konstantin Shvachko
>
> There should be a standard way to indicate that files should be highly replicated, appropriate for files that all nodes will read. This should be settable both on file creation and for already-existing files. Perhaps specifying a particular replication value, like Short.MAX_VALUE, or zero, can be used to signal this. The level should not be constant, but should be relative to the cluster size and network topography. If more nodes are added or if nodes are deleted, the actual replication count should increase or decrease.
> Initially, all that is needed is an API to specify this. It could initially be implemented with a constant (e.g., 10) or with something related to the number of datanodes (sqrt?), and needn't auto-adjust as the cluster size changes That is only the long-term goal.
> When JobClient copies job files (job.xml & job.jar) into the job's filesystem, it should specify this replication level.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-171) need standard API to set dfs
replication = high
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-171?page=comments#action_12376639 ]
eric baldeschwieler commented on HADOOP-171:
--------------------------------------------
highReplicationHint is not the right description IMO. One goal is to optimize on distribution. Call it distributionHint().
Another might be everyRackHint(). Different semantics, different effect.
> need standard API to set dfs replication = high
> -----------------------------------------------
>
> Key: HADOOP-171
> URL: http://issues.apache.org/jira/browse/HADOOP-171
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.2
> Reporter: Doug Cutting
> Assignee: Konstantin Shvachko
>
> There should be a standard way to indicate that files should be highly replicated, appropriate for files that all nodes will read. This should be settable both on file creation and for already-existing files. Perhaps specifying a particular replication value, like Short.MAX_VALUE, or zero, can be used to signal this. The level should not be constant, but should be relative to the cluster size and network topography. If more nodes are added or if nodes are deleted, the actual replication count should increase or decrease.
> Initially, all that is needed is an API to specify this. It could initially be implemented with a constant (e.g., 10) or with something related to the number of datanodes (sqrt?), and needn't auto-adjust as the cluster size changes That is only the long-term goal.
> When JobClient copies job files (job.xml & job.jar) into the job's filesystem, it should specify this replication level.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-171) need standard API to set dfs
replication = high
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-171?page=all ]
eric baldeschwieler resolved HADOOP-171:
----------------------------------------
Resolution: Duplicate
> need standard API to set dfs replication = high
> -----------------------------------------------
>
> Key: HADOOP-171
> URL: http://issues.apache.org/jira/browse/HADOOP-171
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.2
> Reporter: Doug Cutting
> Assignee: Konstantin Shvachko
>
> There should be a standard way to indicate that files should be highly replicated, appropriate for files that all nodes will read. This should be settable both on file creation and for already-existing files. Perhaps specifying a particular replication value, like Short.MAX_VALUE, or zero, can be used to signal this. The level should not be constant, but should be relative to the cluster size and network topography. If more nodes are added or if nodes are deleted, the actual replication count should increase or decrease.
> Initially, all that is needed is an API to specify this. It could initially be implemented with a constant (e.g., 10) or with something related to the number of datanodes (sqrt?), and needn't auto-adjust as the cluster size changes That is only the long-term goal.
> When JobClient copies job files (job.xml & job.jar) into the job's filesystem, it should specify this replication level.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-171) need standard API to set dfs
replication = high
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-171?page=comments#action_12376605 ]
Doug Cutting commented on HADOOP-171:
-------------------------------------
One alternative to 'fs.copyFromLocalFile(localJobJar, remoteJobJar, Short.MAX_VALUE)' might be:
fs.copyFromLocalFile(localJobJar, remoteJobJar);
fs.setReplicatoin(remoteJobJar, Short.MAX_VALUE);
In other words, we indicate this after file creation. Or if folks don't like using Short.MAX_VALUE this way, then this could be something like:
fs.create("job.xml");
fs.setReplicateHighly("job.xml");
fs.copyFromLocalFile(localJobJar, remoteJobJar);
fs.setReplicateHighly(remoteJobJar);
One issue with this is, what would getReplication() return for these? And would setReplication(f, getReplication(f)) be a no-op? An advantage of using a sentinel value like Short.MAX_VALUE is that it doesn't add a lot of special cases to the existing, numeric API.
> need standard API to set dfs replication = high
> -----------------------------------------------
>
> Key: HADOOP-171
> URL: http://issues.apache.org/jira/browse/HADOOP-171
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.2
> Reporter: Doug Cutting
> Assignee: Konstantin Shvachko
>
> There should be a standard way to indicate that files should be highly replicated, appropriate for files that all nodes will read. This should be settable both on file creation and for already-existing files. Perhaps specifying a particular replication value, like Short.MAX_VALUE, or zero, can be used to signal this. The level should not be constant, but should be relative to the cluster size and network topography. If more nodes are added or if nodes are deleted, the actual replication count should increase or decrease.
> Initially, all that is needed is an API to specify this. It could initially be implemented with a constant (e.g., 10) or with something related to the number of datanodes (sqrt?), and needn't auto-adjust as the cluster size changes That is only the long-term goal.
> When JobClient copies job files (job.xml & job.jar) into the job's filesystem, it should specify this replication level.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-171) need standard API to set dfs
replication = high
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-171?page=comments#action_12376622 ]
Doug Cutting commented on HADOOP-171:
-------------------------------------
> We can implement short highReplicationHint()
That would work, but it doesn't seem like the simplest API. But if folks feel strongly I can live with it.
> Then copyFromLocalFile() would use that value to create files.
I don't think we want copyFromLocal to always do that. I think we'd end up with something like:
fs.setReplication(fs.highReplicationHint());
That's okay. If the filesystem grows or shrinks a lot then it might become inadequate. But whatever, if most folks prefer this, I'd be willing to go along.
Perhaps we need to distingush in the API between the requested replication count and the actual replication count. Then one can set the requested level to Small.MAX_VALUE and query the actual value to see how the fs interprets this.
> need standard API to set dfs replication = high
> -----------------------------------------------
>
> Key: HADOOP-171
> URL: http://issues.apache.org/jira/browse/HADOOP-171
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.2
> Reporter: Doug Cutting
> Assignee: Konstantin Shvachko
>
> There should be a standard way to indicate that files should be highly replicated, appropriate for files that all nodes will read. This should be settable both on file creation and for already-existing files. Perhaps specifying a particular replication value, like Short.MAX_VALUE, or zero, can be used to signal this. The level should not be constant, but should be relative to the cluster size and network topography. If more nodes are added or if nodes are deleted, the actual replication count should increase or decrease.
> Initially, all that is needed is an API to specify this. It could initially be implemented with a constant (e.g., 10) or with something related to the number of datanodes (sqrt?), and needn't auto-adjust as the cluster size changes That is only the long-term goal.
> When JobClient copies job files (job.xml & job.jar) into the job's filesystem, it should specify this replication level.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-171) need standard API to set dfs
replication = high
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-171?page=comments#action_12376616 ]
Konstantin Shvachko commented on HADOOP-171:
--------------------------------------------
We can implement
short highReplicationHint()
which would ask namenode before anything what it thinks an appropriate
replication for highly accessed files would be.
Then copyFromLocalFile() would use that value to create files.
The size of the cluster does not vary often. 10% variation due to node
failure and come back is not a big deal with respect to sqrt or /10.
> need standard API to set dfs replication = high
> -----------------------------------------------
>
> Key: HADOOP-171
> URL: http://issues.apache.org/jira/browse/HADOOP-171
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.2
> Reporter: Doug Cutting
> Assignee: Konstantin Shvachko
>
> There should be a standard way to indicate that files should be highly replicated, appropriate for files that all nodes will read. This should be settable both on file creation and for already-existing files. Perhaps specifying a particular replication value, like Short.MAX_VALUE, or zero, can be used to signal this. The level should not be constant, but should be relative to the cluster size and network topography. If more nodes are added or if nodes are deleted, the actual replication count should increase or decrease.
> Initially, all that is needed is an API to specify this. It could initially be implemented with a constant (e.g., 10) or with something related to the number of datanodes (sqrt?), and needn't auto-adjust as the cluster size changes That is only the long-term goal.
> When JobClient copies job files (job.xml & job.jar) into the job's filesystem, it should specify this replication level.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira