[jira] [Commented] (HADOOP-8054) NPE with FilterFileSystem

["Daryn Sharp (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-8054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208545#comment-13208545 ] 

Daryn Sharp commented on HADOOP-8054:
-------------------------------------

Basically, if an instance of {{FilterFileSystem}} uses a different uri (scheme and/or authority) than the embedded fs, the double init made the embedded fs incorrectly report the filtered's uri.  This breaks all kinds of things like cached fs instances, dupping of the fs via its uri, fs statistics, setting of the token service which is based on the uri's authority, etc.  I think this patch didn't break existing tests because RFS is hardcoded to return file:/// instead of the uri passed to init.  This masks the problem that was fixed.

Other {{FilterFileSystem}} instances use the ctor that takes the embedded {{FileSystem}}, which for the aforementioned reason must not be re-inited, in particular a chrooted fs.  {{LocalFileSystem}} is unique in that its ctor instantiates a RLFS with no conf and relies on the double init to set the conf in the RLFS.  I ran into this jira's issue, so I modified {{LocalFileSystem}} instead of {{FilterFileSystem}}, to continue to re-init the RLFS because it would be no worse than before.

W/o seeing the source code, {{ProxyFileSystem}} appears to be instantiating a RLFS with the default ctor and then "neglects" to call {{setConf}} or {{initialize}} because it could "get away with it" when it was wrapped in a {{FilteredFileSystem}}.  As an aside, I think it was a mistake that {{RawLocalFileSystem}} exposed the default ctor, instead of forcing the use of the ctor that takes a conf...

Long story (sorry): for backwards-compatibility, I'll see if propagating {{FilterFileSystem#setConf(conf)}} to the embedded fs will fix the problem.
                
> NPE with FilterFileSystem
> -------------------------
>
>                 Key: HADOOP-8054
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8054
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.1
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Daryn Sharp
>            Priority: Critical
>         Attachments: HADOOP-8054.patch
>
>
> While running Hive tests, I'm seeing the following exception with 0.23.1,
> {noformat}
> ava.lang.NullPointerException
>         at org.apache.hadoop.fs.FileSystem.getDefaultBlockSize(FileSystem.java:1901)
>         at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:447)
>         at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:351)
>         at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:351)
>         at org.apache.hadoop.fs.ProxyFileSystem.getFileStatus(ProxyFileSystem.java:247)
>         at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:351)
>         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1165)
>         at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:390)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:242)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:232)
> {noformat}
> Did not see this with 0.23.0, though.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira