You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Noah Watkins (JIRA)" <ji...@apache.org> on 2011/01/11 22:14:46 UTC
[jira] Created: (HADOOP-7099) Make RawLocalFileSystem more friendly
to sub-classing
Make RawLocalFileSystem more friendly to sub-classing
-----------------------------------------------------
Key: HADOOP-7099
URL: https://issues.apache.org/jira/browse/HADOOP-7099
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 0.23.0
Reporter: Noah Watkins
Fix For: 0.23.0
This patch does 2 things that makes sub-classing RawLocalFileSystem easier.
First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
The second thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7099) Make RawLocalFileSystem more friendly
to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noah Watkins updated HADOOP-7099:
---------------------------------
Attachment: HADOOP-7099.patch
Please review.
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980867#action_12980867 ]
Owen O'Malley commented on HADOOP-7099:
---------------------------------------
Actually, this will help fix another annoyance where the RawLocal and Local FileSystem both use "file:" as their protocol. We probably should use "file:" for raw and "filecrc:" for checksummed.
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980344#action_12980344 ]
Noah Watkins commented on HADOOP-7099:
--------------------------------------
Thanks for the feedback. I think that FilterFileSystem will work; I'll do some testing with it to see. As an alternative, would you be opposed to a patch that started from a duplicate copy of RawLocalFileSystem? In a lot of ways that might be the best solution, especially thinking forward to cases in which the two diverge. But, I want to check if there is any major issue with that code duplication?
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981055#action_12981055 ]
Noah Watkins commented on HADOOP-7099:
--------------------------------------
Hi Owen. Would you like to see a patch to FilterFileSystem now, or should I instead perform these path conversions in my own file system class?
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980442#action_12980442 ]
Owen O'Malley commented on HADOOP-7099:
---------------------------------------
Couldn't you just override the checkPath implementation in FilterFileSystem? It defaults to passing to the lower level, but clearly you don't want it to.
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7099) Make RawLocalFileSystem more friendly
to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noah Watkins updated HADOOP-7099:
---------------------------------
Tags: RawLocalFileSystem FileSystem (was: RawLocalFileSystem)
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Fix For: 0.23.0
>
>
> This patch does 2 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> The second thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980402#action_12980402 ]
Noah Watkins commented on HADOOP-7099:
--------------------------------------
Yes, this is for Ceph. The solution we are using is great: code is identical to RawLocalFileSystem, but for files located under a Ceph mount point, getBlockLocations consults the in-kernel client via an IOCTL call written in JNI. That's the high-level view. In practice it has been a little hard to get right in terms of something that fits well into the Hadoop code base.
So, the fundamental issue is that access to Ceph is through the local file system (i.e. file:///), but for a subset of the file system hierarchy (i.e. the Ceph mount point), a different file system implementation should be used, not just for accounting, but also for altered functionality (i.e. the ioctl availability for Ceph files). We use a configuration parameter to specify the mount point, and force access through the Hadoop Ceph File System to stay within the mount point.
With sub-classing, the solution is nice because changing the URI (ceph:///) doesn't cause the problem I cited in the previous comment (checkPath blowing up).
Does that paint a clearer picture? We now have our code working well, but are struggling with how to package it up for distribution.
Thanks!
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980362#action_12980362 ]
Noah Watkins commented on HADOOP-7099:
--------------------------------------
Yes, sounds good. There is a minor change to RawLocal that will be needed, but I'lll start a new JIRA for that. It's unrelated to this JIRA. Thanks a lot!
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980850#action_12980850 ]
Noah Watkins commented on HADOOP-7099:
--------------------------------------
Owen, this all sounds great. I'll move forward then with transforming my inheritance version to use delegation.
Do you have any thoughts on possible points in the abstraction that might "leak" paths that contain the nested file system's URI scheme? Should there also be a hook in FilterFileSystem that will change the scheme of any path returned from the nested class?
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7099) Make RawLocalFileSystem more friendly
to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noah Watkins updated HADOOP-7099:
---------------------------------
Priority: Minor (was: Major)
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7099) Make RawLocalFileSystem more friendly
to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noah Watkins updated HADOOP-7099:
---------------------------------
Description:
This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
Second, allows subclasses to modify the working directory.
The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
was:
This patch does 2 things that makes sub-classing RawLocalFileSystem easier.
First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
The second thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Fix For: 0.23.0
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980416#action_12980416 ]
Noah Watkins commented on HADOOP-7099:
--------------------------------------
Explained more succinctly:
We want to reuse the RawLocalFileSystem code (plus some additions, and new URI scheme):
- Sub-classing seems to have caused problems in the past
- Delegation with FIlterFileSystem poses issues with checkPath
- And duplicating the code just seems inefficient.
:) Thanks for the patience in explaining this well.
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7099) Make RawLocalFileSystem more friendly
to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noah Watkins updated HADOOP-7099:
---------------------------------
Status: Patch Available (was: Open)
Please review.
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980339#action_12980339 ]
Todd Lipcon commented on HADOOP-7099:
-------------------------------------
I haven't looked at your patch, but in general I think subclassing is a very fragile mechanism for providing new filesystems, so not sure if we should be making changes specifically for subclassers. We fell into this trap for a year or two in HBase development and it only came back to bite us hard later. FilterFileSystem is a much more maintainable mechanism. Does it fit the bill for your use case?
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980358#action_12980358 ]
Todd Lipcon commented on HADOOP-7099:
-------------------------------------
Cool, shall we resolve this one as wontfix then?
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7099) Make RawLocalFileSystem more friendly
to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noah Watkins updated HADOOP-7099:
---------------------------------
Status: Open (was: Patch Available)
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980389#action_12980389 ]
Noah Watkins commented on HADOOP-7099:
--------------------------------------
Well, I'm afraid that using the FilterFileSystem does pose a little bit of a road block. It has to do with checkPath failing when the FilteredFileSystem changes its scheme. So, for example, if a FilteredFileSystem uses RawLocalFileSystem as the base FS, but changes its scheme to be "new-scheme", then RawLocalFileSystem#checkPath fails because it expects "file" scheme.
In this case is it better to design a new checkPath that lets "new-scheme" through, or is this indicative of a larger problem?
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980451#action_12980451 ]
Noah Watkins commented on HADOOP-7099:
--------------------------------------
It doesn't seem to be FilterFileSystem#checkPath that is the problem, but rather RawLocalFileSystem#checkPath.
Using Ceph as an example, in which CephFileSystem extends FilterFileSystem:
- If a path exists in CephFileSystem, it will have scheme "ceph"
- When "delete()" is called the path is passed down to RawLocalFileSystem from CephFileSystem.
- However, once in RawLocalFileSystem, any call to checkPath will expect the "file" scheme, and blow up.
This problem doesn't occur in the current setup because classes using FilterFileSystem to delegate to a lower FS use the same scheme.
One way to solve this in FilterFileSystem might be to transform every path that passes through to be of the "file" scheme that RawLocal expects. However, this seems likely quite fragile. Thoughts?
Thanks
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980391#action_12980391 ]
Todd Lipcon commented on HADOOP-7099:
-------------------------------------
Hmm, that's a good point. Owen, I noticed you're watching this ticket, do you have any thoughts here? Noah, maybe you can elaborate a little on the use case? Is this for Ceph?
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980845#action_12980845 ]
Owen O'Malley commented on HADOOP-7099:
---------------------------------------
Ah, of course. You'll need to re-write the path to have a file: schema before you delegate it down. In the long term, it probably makes sense to have support for that in FilterFileSystem. Notably:
{code}
protected Path modifyPathForInnerFileSystem(Path path) throws IOException {
return path;
}
{code}
And then in FilterFileSystem all of the default bodies should call that method to transform the path before passing the path to the nested file system.
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-7099:
----------------------------
Fix Version/s: (was: 0.24.0)
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Noah Watkins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980357#action_12980357 ]
Noah Watkins commented on HADOOP-7099:
--------------------------------------
Yes, I've seen the new FS interfaces, but I'm waiting for a new implementation of RawLocal that uses it (I believe the current version uses DelegateToFileSystem?). This all sounds good. I'll switch over to FilterFileSystem for now. Thanks!
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7099) Make RawLocalFileSystem more
friendly to sub-classing
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980350#action_12980350 ]
Todd Lipcon commented on HADOOP-7099:
-------------------------------------
I wouldn't want to duplicate the class inside Hadoop - the goal is to keep the core small where possible, I think, especially where we're talking about new interfaces we'll have to support.
Down the road we already have a new parallel set of classes for FS access on the way - FileContext and the "new" FileSystem. Worth checking out if you haven't seen them yet.
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling getInitialWorkingDirectory(). This is important because if a sub-class has an initially null uri (prior to initialize() being called), then getInitialWorkingDirectory() will cause a NullPointerException when it tries to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI object to the File(URI) constructor, but rather pass the string representation of the path. This is important because URI's that are not using the "file" scheme will cause the File(URI) constructor to throw an exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.