You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Ruyue Ma (JIRA)" <ji...@apache.org> on 2008/12/03 18:39:44 UTC
[jira] Created: (HADOOP-4764) add replication factor for hdfs
directory
add replication factor for hdfs directory
-----------------------------------------
Key: HADOOP-4764
URL: https://issues.apache.org/jira/browse/HADOOP-4764
Project: Hadoop Core
Issue Type: New Feature
Components: dfs
Reporter: Ruyue Ma
If we can set replication factor for directory. we can modify the DFSClent.create() method, pass 0 for the default block replication. Namenode check create request, if blockreplication is 0, it will give its parent dir replication factor to the file blockreplication factor. This will simplify the administration work. You know we can set /Test or /Tmp dir's replication factor 2 or 1, then all their children files and dirs replication factor is 2 or 1 defaultly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4764) add replication factor for hdfs
directory
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653956#action_12653956 ]
Konstantin Shvachko commented on HADOOP-4764:
---------------------------------------------
Ruyue Ma,
# Currently recursive setReplication() for a directory is handled on the client side. The name-node can change replication only for individual files. So I was proposing to optimize that by handling recursive replications on the name-node itself.
# I do not understand your proposal to change only the edits file format. Does that mean fsimage will not store replication for directories? If so then after second name-node restart you loose all replications for current directories, right?
# I did not get exactly what is the semantics of directory replication: does it apply to direct offsprings only or the whole subtree?
Zheng Shao,
Which part of the source code will be responsible for reading the dummy file and retrieving the default replication? If it is done on the application level, I am fine with it. If the name-node will have to create an extra file for every directory then I am worried a lot.
Adding additional fields to file/directory inodes or creating additional "system" files increases the memory footprint of the namespace (will require larger heap size to store the same number of files). With directories it is not so critical, because there is not so many of them, but still should be justified.
Based on my estimation of required code changes I should mention that what you propose (introducing replications for directories) is a new feature, and should go through all the steps required for new features, which include
- motivation;
- design document;
- implementation (patch);
- test planning, testing, and support.
So if you feel like you have enough bandwidth, enthusiasm, etc to do that, lets start with the design proposal.
> add replication factor for hdfs directory
> -----------------------------------------
>
> Key: HADOOP-4764
> URL: https://issues.apache.org/jira/browse/HADOOP-4764
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Ruyue Ma
> Assignee: Ruyue Ma
>
> If we can set replication factor for directory. we can modify the DFSClent.create() method, pass 0 for the default block replication. Namenode check create request, if blockreplication is 0, it will give its parent dir replication factor to the file blockreplication factor. This will simplify the administration work. You know we can set /Test or /Tmp dir's replication factor 2 or 1, then all their children files and dirs replication factor is 2 or 1 defaultly.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4764) add replication factor for hdfs
directory
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652894#action_12652894 ]
Konstantin Shvachko commented on HADOOP-4764:
---------------------------------------------
This may become a big patch, as e.g. the new field will require change of the image format.
Would it be sufficient to extend implementation of {{setReplication()}} so that it recursively changed replication for all files in the directory subtree?
> add replication factor for hdfs directory
> -----------------------------------------
>
> Key: HADOOP-4764
> URL: https://issues.apache.org/jira/browse/HADOOP-4764
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Ruyue Ma
>
> If we can set replication factor for directory. we can modify the DFSClent.create() method, pass 0 for the default block replication. Namenode check create request, if blockreplication is 0, it will give its parent dir replication factor to the file blockreplication factor. This will simplify the administration work. You know we can set /Test or /Tmp dir's replication factor 2 or 1, then all their children files and dirs replication factor is 2 or 1 defaultly.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-4764) add replication factor
for hdfs directory
Posted by "Ruyue Ma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653153#action_12653153 ]
mry.maillist edited comment on HADOOP-4764 at 12/4/08 11:19 PM:
------------------------------------------------------------
Hi, Konstantin Shvachko
Current trunk version has implemented what you said.
But it is different from my point. You know the default replication factor for file in hadoop-default.xml is 3. few people will modify it. And if i (as a hadoop administrator) want to change the default value, want to limit some file replication factor in some directory to 2. I don't know how to do now. Current hadoop can't support it.
If i can set replication factor for directory, the file's default replication factor will inherite its parent dir's factor. I can give different dir different default replication factor.
Yes it is a big patch. it will modify the FSEditLog format. But it can be compatible with old format.
Please forgive me for my poor English!
Thanks
was (Author: mry.maillist):
Hi, Konstantin Shvachko
Current trunk version has implemented what you said.
But it is different from my point. You know the default replication factor for file in hadoop-default.xml is 3. few people will modify it. And if i (as a hadoop administrator) want to change the default value, want to limit some file replication factor in some directory to 2. I don't know how to do now. Current hadoop can't support it.
If i can set replication factor for directory, the file's default replication factor will inherite its parent dir's factor. I can give different dir different default replication factor.
Yes it is a big patch. it will modify the image format. But it can be compatible with old format.
Please forgive me for my poor English!
Thanks
> add replication factor for hdfs directory
> -----------------------------------------
>
> Key: HADOOP-4764
> URL: https://issues.apache.org/jira/browse/HADOOP-4764
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Ruyue Ma
> Assignee: Ruyue Ma
>
> If we can set replication factor for directory. we can modify the DFSClent.create() method, pass 0 for the default block replication. Namenode check create request, if blockreplication is 0, it will give its parent dir replication factor to the file blockreplication factor. This will simplify the administration work. You know we can set /Test or /Tmp dir's replication factor 2 or 1, then all their children files and dirs replication factor is 2 or 1 defaultly.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4764) add replication factor for hdfs
directory
Posted by "Ruyue Ma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652869#action_12652869 ]
Ruyue Ma commented on HADOOP-4764:
----------------------------------
I have completed the patch, I will submit it as soon as possible
> add replication factor for hdfs directory
> -----------------------------------------
>
> Key: HADOOP-4764
> URL: https://issues.apache.org/jira/browse/HADOOP-4764
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Ruyue Ma
>
> If we can set replication factor for directory. we can modify the DFSClent.create() method, pass 0 for the default block replication. Namenode check create request, if blockreplication is 0, it will give its parent dir replication factor to the file blockreplication factor. This will simplify the administration work. You know we can set /Test or /Tmp dir's replication factor 2 or 1, then all their children files and dirs replication factor is 2 or 1 defaultly.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4764) add replication factor for hdfs
directory
Posted by "Ruyue Ma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653153#action_12653153 ]
Ruyue Ma commented on HADOOP-4764:
----------------------------------
Hi, Konstantin Shvachko
Current trunk version has implemented what you said.
But it is different from my point. You know the default replication factor for file in hadoop-default.xml is 3. few people will modify it. And if i (as a hadoop administrator) want to change the default value, want to limit some file replication factor in some directory to 2. I don't know how to do now. Current hadoop can't support it.
If i can set replication factor for directory, the file's default replication factor will inherite its parent dir's factor. I can give different dir different default replication factor.
Yes it is a big patch. it will modify the image format. But it can be compatible with old format.
Please forgive me for my poor English!
Thanks
> add replication factor for hdfs directory
> -----------------------------------------
>
> Key: HADOOP-4764
> URL: https://issues.apache.org/jira/browse/HADOOP-4764
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Ruyue Ma
>
> If we can set replication factor for directory. we can modify the DFSClent.create() method, pass 0 for the default block replication. Namenode check create request, if blockreplication is 0, it will give its parent dir replication factor to the file blockreplication factor. This will simplify the administration work. You know we can set /Test or /Tmp dir's replication factor 2 or 1, then all their children files and dirs replication factor is 2 or 1 defaultly.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4764) add replication factor for hdfs
directory
Posted by "Ruyue Ma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653666#action_12653666 ]
Ruyue Ma commented on HADOOP-4764:
----------------------------------
We doesn't need modify the FSImage, we only modify the FSEditLog.
Current FSImage sets 0 to the default replication factor for directory.
The modification for FSEditLog doesn't affect the compatible with old log format.
the original code in FSEditLog
/**
* Add create directory record to edit log
*/
public void logMkDir(String path, INode newNode) {
UTF8 info[] = new UTF8[] {
new UTF8(path),
FSEditLog.toLogLong(newNode.getModificationTime()),
FSEditLog.toLogLong(newNode.getAccessTime())
};
logEdit(OP_MKDIR, new ArrayWritable(UTF8.class, info),
newNode.getPermissionStatus());
}
is modified to the following:
/**
* Add create directory record to edit log
*/
public void logMkDir(String path, INode newNode) {
UTF8 info[] = new UTF8[] {
new UTF8(path),
FSEditLog.toLogReplication(newNode.getReplication()),
FSEditLog.toLogLong(newNode.getModificationTime()),
FSEditLog.toLogLong(newNode.getAccessTime())
};
logEdit(OP_MKDIR, new ArrayWritable(UTF8.class, info),
newNode.getPermissionStatus());
}
> add replication factor for hdfs directory
> -----------------------------------------
>
> Key: HADOOP-4764
> URL: https://issues.apache.org/jira/browse/HADOOP-4764
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Ruyue Ma
> Assignee: Ruyue Ma
>
> If we can set replication factor for directory. we can modify the DFSClent.create() method, pass 0 for the default block replication. Namenode check create request, if blockreplication is 0, it will give its parent dir replication factor to the file blockreplication factor. This will simplify the administration work. You know we can set /Test or /Tmp dir's replication factor 2 or 1, then all their children files and dirs replication factor is 2 or 1 defaultly.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4764) add replication factor for hdfs
directory
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653664#action_12653664 ]
Zheng Shao commented on HADOOP-4764:
------------------------------------
So the modification of the image format is the extra default replication factor information of the directory?
Another similar approach is to create a dummy file (e..g, ".default.attribute" under a directory and set the default block replication number on that dummy file.
This won't require modifying the image format. Other changes are similar.
Does that sound better Konstantin?
> add replication factor for hdfs directory
> -----------------------------------------
>
> Key: HADOOP-4764
> URL: https://issues.apache.org/jira/browse/HADOOP-4764
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Ruyue Ma
> Assignee: Ruyue Ma
>
> If we can set replication factor for directory. we can modify the DFSClent.create() method, pass 0 for the default block replication. Namenode check create request, if blockreplication is 0, it will give its parent dir replication factor to the file blockreplication factor. This will simplify the administration work. You know we can set /Test or /Tmp dir's replication factor 2 or 1, then all their children files and dirs replication factor is 2 or 1 defaultly.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4764) add replication factor for hdfs
directory
Posted by "Ruyue Ma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruyue Ma reassigned HADOOP-4764:
--------------------------------
Assignee: Ruyue Ma
we wil give the patch
> add replication factor for hdfs directory
> -----------------------------------------
>
> Key: HADOOP-4764
> URL: https://issues.apache.org/jira/browse/HADOOP-4764
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Ruyue Ma
> Assignee: Ruyue Ma
>
> If we can set replication factor for directory. we can modify the DFSClent.create() method, pass 0 for the default block replication. Namenode check create request, if blockreplication is 0, it will give its parent dir replication factor to the file blockreplication factor. This will simplify the administration work. You know we can set /Test or /Tmp dir's replication factor 2 or 1, then all their children files and dirs replication factor is 2 or 1 defaultly.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.