You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Milind Bhandarkar (JIRA)" <ji...@apache.org> on 2006/12/01 00:05:21 UTC
[jira] Created: (HADOOP-771) Namenode should return error when
trying to delete non-empty directory
Namenode should return error when trying to delete non-empty directory
----------------------------------------------------------------------
Key: HADOOP-771
URL: http://issues.apache.org/jira/browse/HADOOP-771
Project: Hadoop
Issue Type: Improvement
Components: dfs
Affects Versions: 0.8.0
Environment: all
Reporter: Milind Bhandarkar
Currently, the namenode.delete() method allows recursive deletion of a directory. That is, even a non-empty directory could be deleted using namenode.delete(). To avoid costly programmer errors, the namenode should not remove the non-empty directories in this method. Recursively deleting directory should either be performed with listPaths() followed by a delete() for every path, or with a specific namenode method such as deleteRecursive().
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-771) Namenode should return error when
trying to delete non-empty directory
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496994 ]
Milind Bhandarkar commented on HADOOP-771:
------------------------------------------
I agree. The first three fixes should be given a higher priority. But eventually allowing recursive deletion with a single RPC should be prevented. It will also allow us to move the INode inner class outside of FSDirectory, and save on precious memory.
> Namenode should return error when trying to delete non-empty directory
> ----------------------------------------------------------------------
>
> Key: HADOOP-771
> URL: https://issues.apache.org/jira/browse/HADOOP-771
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.8.0
> Environment: all
> Reporter: Milind Bhandarkar
> Assigned To: Sameer Paranjpye
>
> Currently, the namenode.delete() method allows recursive deletion of a directory. That is, even a non-empty directory could be deleted using namenode.delete(). To avoid costly programmer errors, the namenode should not remove the non-empty directories in this method. Recursively deleting directory should either be performed with listPaths() followed by a delete() for every path, or with a specific namenode method such as deleteRecursive().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-771) Namenode should return error when
trying to delete non-empty directory
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12497010 ]
Sameer Paranjpye commented on HADOOP-771:
-----------------------------------------
> If we get rid of recursive deletion then we'll still have a method somewhere that implements recursive deletion and folks will still call it and things will still get unintentionally
> deleted. I don't think that's the solution here.
I agree, even if we prohibit recursive deletion at the Namenode it'll get implemented in a library and called nonetheless. One might be able to abort a deletion in flight, but I don't believe that a user could react quickly enough to a mistaken deletion. The increase in the number of Namenode RPCs is an additional problem.
Hairong's suggestion for a boolean flag specifying recursion would be a good way to go, it would require programmers to be explicit about recursion and avoid additional RPC traffic.
> Namenode should return error when trying to delete non-empty directory
> ----------------------------------------------------------------------
>
> Key: HADOOP-771
> URL: https://issues.apache.org/jira/browse/HADOOP-771
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.8.0
> Environment: all
> Reporter: Milind Bhandarkar
> Assigned To: Sameer Paranjpye
>
> Currently, the namenode.delete() method allows recursive deletion of a directory. That is, even a non-empty directory could be deleted using namenode.delete(). To avoid costly programmer errors, the namenode should not remove the non-empty directories in this method. Recursively deleting directory should either be performed with listPaths() followed by a delete() for every path, or with a specific namenode method such as deleteRecursive().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-771) Namenode should return error when
trying to delete non-empty directory
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12497009 ]
Milind Bhandarkar commented on HADOOP-771:
------------------------------------------
>> On the other hand, won't an RPC per file cause a lot of namenode traffic?
yes, it would. But only when user intentionally deletes all his/her data, which I believe happens rarely.
I think, DFS is optimized for a small number of large files, rather than large number of small files, so in practice, this erasure of entire tree should not happen frequently.
> Namenode should return error when trying to delete non-empty directory
> ----------------------------------------------------------------------
>
> Key: HADOOP-771
> URL: https://issues.apache.org/jira/browse/HADOOP-771
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.8.0
> Environment: all
> Reporter: Milind Bhandarkar
> Assigned To: Sameer Paranjpye
>
> Currently, the namenode.delete() method allows recursive deletion of a directory. That is, even a non-empty directory could be deleted using namenode.delete(). To avoid costly programmer errors, the namenode should not remove the non-empty directories in this method. Recursively deleting directory should either be performed with listPaths() followed by a delete() for every path, or with a specific namenode method such as deleteRecursive().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-771) Namenode should return error when
trying to delete non-empty directory
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496982 ]
Milind Bhandarkar commented on HADOOP-771:
------------------------------------------
>> think the problem is that 'FileSystem#remove(new Path(""))' removes one's home directory.
I think that's just one part of the problem.
A single RPC to delete an entire directory structure is the main problem, I think. If deleting a directory were to take multiple roundtrips between client and namenode, then it would be possible to ^C the client to stop wiping out the entire tree. No ?
> Namenode should return error when trying to delete non-empty directory
> ----------------------------------------------------------------------
>
> Key: HADOOP-771
> URL: https://issues.apache.org/jira/browse/HADOOP-771
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.8.0
> Environment: all
> Reporter: Milind Bhandarkar
> Assigned To: Sameer Paranjpye
>
> Currently, the namenode.delete() method allows recursive deletion of a directory. That is, even a non-empty directory could be deleted using namenode.delete(). To avoid costly programmer errors, the namenode should not remove the non-empty directories in this method. Recursively deleting directory should either be performed with listPaths() followed by a delete() for every path, or with a specific namenode method such as deleteRecursive().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-771) Namenode should return error when
trying to delete non-empty directory
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496995 ]
Hairong Kuang commented on HADOOP-771:
--------------------------------------
Currently FileSystem APIs only support "rmr". To avoid programmatic error, should we change the FileSystem interface so that it distinguishes between rm and rmr? Instead of having "void delete(Path)", we could have "void delete(Path, boolean)". The boolean flag indicates if the deletion is recursive or not.
> Namenode should return error when trying to delete non-empty directory
> ----------------------------------------------------------------------
>
> Key: HADOOP-771
> URL: https://issues.apache.org/jira/browse/HADOOP-771
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.8.0
> Environment: all
> Reporter: Milind Bhandarkar
> Assigned To: Sameer Paranjpye
>
> Currently, the namenode.delete() method allows recursive deletion of a directory. That is, even a non-empty directory could be deleted using namenode.delete(). To avoid costly programmer errors, the namenode should not remove the non-empty directories in this method. Recursively deleting directory should either be performed with listPaths() followed by a delete() for every path, or with a specific namenode method such as deleteRecursive().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-771) Namenode should return error when
trying to delete non-empty directory
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496759 ]
Milind Bhandarkar commented on HADOOP-771:
------------------------------------------
Can this be fixed ?
A programmer error can cost a lot here.
No other file system allows this, IMHO for a reason.
Having been burnt once too many times, I would like to disallow recursive delete completely.
> Namenode should return error when trying to delete non-empty directory
> ----------------------------------------------------------------------
>
> Key: HADOOP-771
> URL: https://issues.apache.org/jira/browse/HADOOP-771
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.8.0
> Environment: all
> Reporter: Milind Bhandarkar
> Assigned To: Sameer Paranjpye
>
> Currently, the namenode.delete() method allows recursive deletion of a directory. That is, even a non-empty directory could be deleted using namenode.delete(). To avoid costly programmer errors, the namenode should not remove the non-empty directories in this method. Recursively deleting directory should either be performed with listPaths() followed by a delete() for every path, or with a specific namenode method such as deleteRecursive().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-771) Namenode should return error when
trying to delete non-empty directory
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496960 ]
Doug Cutting commented on HADOOP-771:
-------------------------------------
If we get rid of recursive deletion then we'll still have a method somewhere that implements recursive deletion and folks will still call it and things will still get unintentionally deleted. I don't think that's the solution here.
I think the problem is that 'FileSystem#remove(new Path(""))' removes one's home directory. To address this we could simply prohibit the empty string as a Path. One should be forced to use at least "." to refer to the connected/default directory. Would that work?
> Namenode should return error when trying to delete non-empty directory
> ----------------------------------------------------------------------
>
> Key: HADOOP-771
> URL: https://issues.apache.org/jira/browse/HADOOP-771
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.8.0
> Environment: all
> Reporter: Milind Bhandarkar
> Assigned To: Sameer Paranjpye
>
> Currently, the namenode.delete() method allows recursive deletion of a directory. That is, even a non-empty directory could be deleted using namenode.delete(). To avoid costly programmer errors, the namenode should not remove the non-empty directories in this method. Recursively deleting directory should either be performed with listPaths() followed by a delete() for every path, or with a specific namenode method such as deleteRecursive().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-771) Namenode should return error when
trying to delete non-empty directory
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12497005 ]
Doug Cutting commented on HADOOP-771:
-------------------------------------
> A single RPC to delete an entire directory structure is the main problem
On the other hand, won't an RPC per file cause a lot of namenode traffic?
> Namenode should return error when trying to delete non-empty directory
> ----------------------------------------------------------------------
>
> Key: HADOOP-771
> URL: https://issues.apache.org/jira/browse/HADOOP-771
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.8.0
> Environment: all
> Reporter: Milind Bhandarkar
> Assigned To: Sameer Paranjpye
>
> Currently, the namenode.delete() method allows recursive deletion of a directory. That is, even a non-empty directory could be deleted using namenode.delete(). To avoid costly programmer errors, the namenode should not remove the non-empty directories in this method. Recursively deleting directory should either be performed with listPaths() followed by a delete() for every path, or with a specific namenode method such as deleteRecursive().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-771) Namenode should return error when
trying to delete non-empty directory
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496987 ]
Doug Cutting commented on HADOOP-771:
-------------------------------------
There were a number of things that could have prevented this. First, I'm told this was triggered by a buggy script parser that parsed 'rm<space><space>foo' as 'rm(""); foo();'. Fixing that would avoid this. Second, the script execution engine should have used Hadoop's Trash feature, so that deletions go through an intermediate directory. Third, rm("") shouldn't delete one's home directory but instead throw an exception. And, finally, fourth, if we distinguished between 'rm' and 'rmr', this might not have happened, but there's a good chance that the script would have instead contained 'rmr<space><space>foo', and it still would have happened. So I think distinguishing between recursive and non-recursive file deletion is the weakest defense against this. It might not be a bad thing. Hadoop's own FsShell already makes this distinction. Perhaps Pig's scripting language should too, and perhaps Hadoop's APIs should even make this distinction. But I'd give the first three fixes much higher priority.
> Namenode should return error when trying to delete non-empty directory
> ----------------------------------------------------------------------
>
> Key: HADOOP-771
> URL: https://issues.apache.org/jira/browse/HADOOP-771
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.8.0
> Environment: all
> Reporter: Milind Bhandarkar
> Assigned To: Sameer Paranjpye
>
> Currently, the namenode.delete() method allows recursive deletion of a directory. That is, even a non-empty directory could be deleted using namenode.delete(). To avoid costly programmer errors, the namenode should not remove the non-empty directories in this method. Recursively deleting directory should either be performed with listPaths() followed by a delete() for every path, or with a specific namenode method such as deleteRecursive().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-771) Namenode should return error when
trying to delete non-empty directory
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496971 ]
Jim Kellerman commented on HADOOP-771:
--------------------------------------
Doug Cutting wrote:
> I think the problem is that 'FileSystem#remove(new Path(""))' removes one's home directory. To address
> this we could simply prohibit the empty string as a Path. One should be forced to use at least "." to refer
> to the connected/default directory. Would that work?
+1
> Namenode should return error when trying to delete non-empty directory
> ----------------------------------------------------------------------
>
> Key: HADOOP-771
> URL: https://issues.apache.org/jira/browse/HADOOP-771
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.8.0
> Environment: all
> Reporter: Milind Bhandarkar
> Assigned To: Sameer Paranjpye
>
> Currently, the namenode.delete() method allows recursive deletion of a directory. That is, even a non-empty directory could be deleted using namenode.delete(). To avoid costly programmer errors, the namenode should not remove the non-empty directories in this method. Recursively deleting directory should either be performed with listPaths() followed by a delete() for every path, or with a specific namenode method such as deleteRecursive().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.