You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Sameer Paranjpye (JIRA)" <ji...@apache.org> on 2006/06/07 01:06:29 UTC
[jira] Created: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
dfs.FSDirectory.mkdirs can create sub-directories of a file!
------------------------------------------------------------
Key: HADOOP-281
URL: http://issues.apache.org/jira/browse/HADOOP-281
Project: Hadoop
Type: Bug
Versions: 0.3.1
Reporter: Sameer Paranjpye
dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=comments#action_12421668 ]
Arun C Murthy commented on HADOOP-281:
--------------------------------------
Hi Wendy,
Your approach looks fine too... I'll just let Doug/Konstantin or someone else with more experience with dfs codebase decide.
However, a subtle issue: there are some places in the code base (e.g. FileUtil.java) which makes a call to 'mkdirs' and then fail to check the return-value to ensure that it completed successfully, which lead to other bugs. At the very least we will need to patch FileUtil.java (it's a part of my mkdirs.patch). Could you further test your patch using the dfs shell?
E.g.
$ hadoop dfs -mkdir /tmp/test-mkdir
$ hadoop dfs -copyFromLocal ./foo.txt /tmp/test-mkdir/foo.txt
$ hadoop dfs -mkdir /tmp/test-mkdir/foo.txt/d1/d2
$ hadoop dfs -mkdir /tmp/test-mkdir/foo.txt/d1
$ hadoop dfs -put ./some_directory /tmp/test-mkdir/foo.txt/some_directory
Your patch fails only in the last test-case above (due to non-existent return-value check in FileUtil.java).
Could you please take a look and maybe include my patch for FileUtil.java too?
thanks,
Arun
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Fix For: 0.5.0
>
> Attachments: addNode.patch, mkdirs.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=all ]
Doug Cutting updated HADOOP-281:
--------------------------------
Fix Version: 0.5.0
(was: 0.4.0)
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Type: Bug
> Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Fix For: 0.5.0
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=all ]
Arun C Murthy updated HADOOP-281:
---------------------------------
Attachment: mkdirs.patch
I have attached a fairly simple patch which essentially checks to ensure that all existing components of the path are directories by checking while 'collecting' the parents in dfs.FSDirectory.mkdirs.
This patch also necessiated (straightforward) changes to a few places where the return-value of 'mkdirs' wasn't being checked...
thanks,
Arun
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Fix For: 0.5.0
>
> Attachments: mkdirs.patch
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=comments#action_12422504 ]
Konstantin Shvachko commented on HADOOP-281:
--------------------------------------------
We can still return the correct boolean value to preserve public interface,
but throw an exception at the same time. The prototype is in place.
Or we can use a usual "trick" of deprecating the old mkdirs() , and define a
new one that is void.
Which mkdirs() is called by applications?
I believe mkdirs() is public since it is a part of the ClientProtocol interface.
There are other mkdirs() further in the NameNode data structure implementation.
Those should not be and are not called by anything outside the dfs project.
These mkdirs() don't need to be public at all, imo.
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Fix For: 0.5.0
>
> Attachments: addNode.patch, mkdirs.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=comments#action_12421706 ]
Konstantin Shvachko commented on HADOOP-281:
--------------------------------------------
I think addNode() is a good place to check whether the parent is a directory.
mkdirs() does not need to check all parents in the path, just the last one, since
all other parents have already been checked before.
I agree with Arun, the return value of mkdirs() should be checked every time
it is called to make sure the directories were actually created, including FileUtil
copy methods.
For mkdirs() I would let the FileNotFoundException exception thrown by addNode()
go through, rather than catching it and returning boolean, so that whoever calls
mkdirs() could decide what to do.
But there might be different opinions on that, and this will require more code changes.
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Fix For: 0.5.0
>
> Attachments: addNode.patch, mkdirs.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=comments#action_12430987 ]
Benjamin Reed commented on HADOOP-281:
--------------------------------------
Have we considered this to be a feature. Some newer general purpose file system allow a file to also be a directory (and visa versa). It is an abstraction that nicely folds in the functionality of streams and resource forks. For example, rather than sticking crcs and other attributes for a file in the same directory as a file (which necessitates a bunch of ugly skip logic in the code), we could simply put them in a subdirectory of the file.
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Assigned To: Wendy Chien
> Fix For: 0.6.0
>
> Attachments: addNode.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=comments#action_12422458 ]
eric baldeschwieler commented on HADOOP-281:
--------------------------------------------
Why not throw an exception on failure? That seems like the more correct thing to do.
Checking return codes for failure is so 20th century.
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Fix For: 0.5.0
>
> Attachments: addNode.patch, mkdirs.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=all ]
Wendy Chien updated HADOOP-281:
-------------------------------
Attachment: (was: mkdirs.patch)
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Assigned To: Wendy Chien
> Fix For: 0.6.0
>
> Attachments: addNode.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=all ]
Wendy Chien updated HADOOP-281:
-------------------------------
Attachment: addNode.patch
TestDFSMkdirs.java
Hi Arun,
I was simultaneously working on this bug and had a different fix. Hairong and I think it would be simpler to check the path/parent is a directory in addNode (which is called by unprotectedMkdir which in turn is called by mkdirs). I think the check in addNode is required regardless of mkdirs.
I have attached my patch file along with test code for the change.
Please look at it and let me know what you think.
Thanks,
Wendy
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Fix For: 0.5.0
>
> Attachments: addNode.patch, mkdirs.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=all ]
Doug Cutting updated HADOOP-281:
--------------------------------
Status: Resolved (was: Patch Available)
Resolution: Fixed
I just committed this. Thanks, Wendy!
Ben: yes, it would be a feature, but a feature with many bugs. There are lots of places in the current API that assume files are distinct from directories. So, perhaps, in a separate issue, we could discuss adding this as a feature (and how we'd implement it on linux and windows), but, for now, the most consistent thing to do is treat it as a bug.
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Assigned To: Wendy Chien
> Fix For: 0.6.0
>
> Attachments: addNode.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=comments#action_12421733 ]
Wendy Chien commented on HADOOP-281:
------------------------------------
Yes, the callers of mkdirs will need to check the return value. I looked over Arun's changes to FileUtil and they seem fine. I can make a separate patch which includes those changes once we decide if mkdirs should pass along the FilenotFoundException or only return a boolean.
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Fix For: 0.5.0
>
> Attachments: addNode.patch, mkdirs.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=all ]
Wendy Chien updated HADOOP-281:
-------------------------------
Status: Patch Available (was: Open)
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Assigned To: Wendy Chien
> Fix For: 0.6.0
>
> Attachments: addNode.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=all ]
Sameer Paranjpye updated HADOOP-281:
------------------------------------
Fix Version: 0.4.0
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Type: Bug
> Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Fix For: 0.4.0
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Assigned: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=all ]
Sameer Paranjpye reassigned HADOOP-281:
---------------------------------------
Assignee: Wendy Chien
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Assigned To: Wendy Chien
> Fix For: 0.6.0
>
> Attachments: addNode.patch, mkdirs.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-281) dfs.FSDirectory.mkdirs can create
sub-directories of a file!
Posted by "Yoram Arnon (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-281?page=comments#action_12422485 ]
Yoram Arnon commented on HADOOP-281:
------------------------------------
throwing an exception was suggested, but rejected in order not to change a public interface that may be (is) used by applications.
> dfs.FSDirectory.mkdirs can create sub-directories of a file!
> ------------------------------------------------------------
>
> Key: HADOOP-281
> URL: http://issues.apache.org/jira/browse/HADOOP-281
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.3.1
> Reporter: Sameer Paranjpye
> Fix For: 0.5.0
>
> Attachments: addNode.patch, mkdirs.patch, TestDFSMkdirs.java
>
>
> dfs.FSDirectory.mkdirs will merrily adds children to a directory tree node without checking whether it represents a directory. So it is possible to create a subdirectories of a file.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira