You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2006/11/16 09:08:37 UTC
[jira] Created: (HADOOP-730) Local file system uses copy to
implement rename
Local file system uses copy to implement rename
-----------------------------------------------
Key: HADOOP-730
URL: http://issues.apache.org/jira/browse/HADOOP-730
Project: Hadoop
Issue Type: Bug
Components: fs
Affects Versions: 0.8.0
Reporter: Owen O'Malley
Assigned To: Owen O'Malley
Fix For: 0.9.0
There is a variable LocalFileSystem.useCopyForRename that is set to true. When true, the local file system will implement rename as a copy followed by a delete. This is likely a performance problem. Is there a reason that useCopyForRename is set?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-730) Local file system uses copy to
implement rename
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532173 ]
Doug Cutting commented on HADOOP-730:
-------------------------------------
> Couldn't it try the renameTo() operation and if it fails then fallback to the slower copy&delete method?
Sounds good to me. That might provide a nice speedup too. Care to provide a patch?
> Local file system uses copy to implement rename
> -----------------------------------------------
>
> Key: HADOOP-730
> URL: https://issues.apache.org/jira/browse/HADOOP-730
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.8.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
>
> There is a variable LocalFileSystem.useCopyForRename that is set to true. When true, the local file system will implement rename as a copy followed by a delete. This is likely a performance problem. Is there a reason that useCopyForRename is set?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-730) Local file system uses
copy to implement rename
Posted by "Rod Taylor (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532151 ]
rbt edited comment on HADOOP-730 at 10/3/07 9:12 AM:
------------------------------------------------------------
Couldn't it try the renameTo() operation and if it fails then fallback to the slower copy&delete method?
if (pathToFile(src).renameTo(pathToFile(dst))) {
return true;
}
else {
return FileUtil.copy(this, src, this, dst, true, getConf());
}
would replace
if (useCopyForRename) {
return FileUtil.copy(this, src, this, dst, true, getConf());
} else return pathToFile(src).renameTo(pathToFile(dst));
was (Author: rbt):
Couldn't it try the renameTo() operation and if it fails then fallback to the slower copy&delete method?
if (pathToFile(src).renameTo(pathToFile(dst))) {
return true;
}
else {
return FileUtil.copy(this, src, this, dst, true, getConf());
}
would replace
if (useCopyForRename) {
return FileUtil.copy(this, src, this, dst, true, getConf());
} else return pathToFile(src).renameTo(pathToFile(dst));
> Local file system uses copy to implement rename
> -----------------------------------------------
>
> Key: HADOOP-730
> URL: https://issues.apache.org/jira/browse/HADOOP-730
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.8.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
>
> There is a variable LocalFileSystem.useCopyForRename that is set to true. When true, the local file system will implement rename as a copy followed by a delete. This is likely a performance problem. Is there a reason that useCopyForRename is set?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-730) Local file system uses
copy to implement rename
Posted by "Rod Taylor (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532151 ]
rbt edited comment on HADOOP-730 at 10/3/07 9:12 AM:
------------------------------------------------------------
Couldn't it try the renameTo() operation and if it fails then fallback to the slower copy&delete method?
if (pathToFile(src).renameTo(pathToFile(dst))) {
return true;
}
else {
return FileUtil.copy(this, src, this, dst, true, getConf());
}
would replace
if (useCopyForRename) {
return FileUtil.copy(this, src, this, dst, true, getConf());
} else return pathToFile(src).renameTo(pathToFile(dst));
was (Author: rbt):
Couldn't it try the renameTo() operation and if it fails then fallback to the slower copy&delete method?
if (pathToFile(src).renameTo(pathToFile(dst))) {
return true;
}
else {
return FileUtil.copy(this, src, this, dst, true, getConf());
}
would replace
if (useCopyForRename) {
return FileUtil.copy(this, src, this, dst, true, getConf());
} else return pathToFile(src).renameTo(pathToFile(dst));
> Local file system uses copy to implement rename
> -----------------------------------------------
>
> Key: HADOOP-730
> URL: https://issues.apache.org/jira/browse/HADOOP-730
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.8.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
>
> There is a variable LocalFileSystem.useCopyForRename that is set to true. When true, the local file system will implement rename as a copy followed by a delete. This is likely a performance problem. Is there a reason that useCopyForRename is set?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-730) Local file system uses copy to
implement rename
Posted by "Rod Taylor (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532151 ]
Rod Taylor commented on HADOOP-730:
-----------------------------------
Couldn't it try the renameTo() operation and if it fails then fallback to the slower copy&delete method?
if (pathToFile(src).renameTo(pathToFile(dst))) {
return true;
}
else {
return FileUtil.copy(this, src, this, dst, true, getConf());
}
would replace
if (useCopyForRename) {
return FileUtil.copy(this, src, this, dst, true, getConf());
} else return pathToFile(src).renameTo(pathToFile(dst));
> Local file system uses copy to implement rename
> -----------------------------------------------
>
> Key: HADOOP-730
> URL: https://issues.apache.org/jira/browse/HADOOP-730
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.8.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
>
> There is a variable LocalFileSystem.useCopyForRename that is set to true. When true, the local file system will implement rename as a copy followed by a delete. This is likely a performance problem. Is there a reason that useCopyForRename is set?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-730) Local file system uses copy to
implement rename
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-730?page=comments#action_12450476 ]
Doug Cutting commented on HADOOP-730:
-------------------------------------
I think the issue was that rename isn't guaranteed to work across volumes. Linux 'mv' typically works across volumes these days, by resorting to copying, but I'm not sure what Java's file rename does.
> Local file system uses copy to implement rename
> -----------------------------------------------
>
> Key: HADOOP-730
> URL: http://issues.apache.org/jira/browse/HADOOP-730
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.8.0
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.9.0
>
>
> There is a variable LocalFileSystem.useCopyForRename that is set to true. When true, the local file system will implement rename as a copy followed by a delete. This is likely a performance problem. Is there a reason that useCopyForRename is set?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira