You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Sanjay Radia (JIRA)" <ji...@apache.org> on 2008/12/29 21:56:44 UTC
[jira] Created: (HADOOP-4952) Improved files system interface for
the application writer.
Improved files system interface for the application writer.
-----------------------------------------------------------
Key: HADOOP-4952
URL: https://issues.apache.org/jira/browse/HADOOP-4952
Project: Hadoop Core
Issue Type: Improvement
Affects Versions: 0.21.0
Reporter: Sanjay Radia
Assignee: Sanjay Radia
Currently the FIleSystem interface serves two purposes:
- an application writer's interface for using the Hadoop file system
- a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
- Filesystem interface has a confusing set of methods for the application writer
- We could make it easier to take advantage of the URI file naming
** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
** The proposed scheme (below) simplifies this.
- The client side config can be simplified.
** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
-
** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
** Further one should need very little to configure the client side:
*** Default files system.
*** Block size
*** Replication factor
*** Scheme to class mapping
** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Johan Liesén (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688168#action_12688168 ]
Johan Liesén commented on HADOOP-4952:
--------------------------------------
In the NIO.2 API there is no concept of a working directory. But, paths aren't resolved against the working directory, but rather against any other directory, so it's easy to introduce a working directory concept on the application level.
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-4952:
---------------------------------
Attachment: Files.java
The attached Files.java is a class for the proposed new interface for the application writer.
The FileSystem class will remain for the fs implementor (after some cleanup); it will also be used to implement the
Files class.
None of the methods in the Files class have an implementation yet - the skeleton is to discuss the proposal.
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683142#action_12683142 ]
Doug Cutting commented on HADOOP-4952:
--------------------------------------
> Is the meaning of a working directory different in HDFS from that in the local file system?
Yes. The local filesystem's working directory is global to the JVM and read-only, while Hadoop FileSystem instances each have a working directory that may be changed. For LocalFileSystem, the working directory is initially the JVM's working directory. For HDFS, the working directory is initially the user's home directory in that filesystem (/user/${username}).
Relative paths are resolved against the working directory. So, "foo" refers to a file in the working directory of the default filesystem while "file:foo" refers to a file in the working directory of the local filesystem. The default filesystem is determined from the Configuration in effect.
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Johan Liesén (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683107#action_12683107 ]
Johan Liesén commented on HADOOP-4952:
--------------------------------------
I do not fully grok the problem here. Is the meaning of a working directory different in HDFS from that in the local file system?
I've just briefly browsed through the NIO.2 FS API and, as far as I can tell, the notion of working directory doesn't really exist. It is only applicable to file systems where the environment (working directory, or similar) is implicitly available (or if there is none). In my opinion, this is a good thing.
In this respect: Paths.get("hdfs://...") doesn't make much sense because the environment for HDFS, the Configuration object, is not available nor is a FileSystem reference.
Wrt. 3. I'm not sure I understand the meaning of "slash relative names"; isn't that an absolute path?
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659659#action_12659659 ]
Tom White commented on HADOOP-4952:
-----------------------------------
Wouldn't it make more sense to keep the FileSystem interface for clients and evolve it, and move the implementor's interface to an SPI?
Also, instead of creating a new interface, we might be able to use the new FileSystem interface being developed in JSR 203. See HADOOP-3518.
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Johan Liesén (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695328#action_12695328 ]
Johan Liesén commented on HADOOP-4952:
--------------------------------------
Yes. I'm not saying that the API defined in JSR 203 maps onto HDFS perfectly.
So, after proposing this as a GSOC in https://issues.apache.org/jira/browse/HADOOP-3518 and e-mailing some back and forth with Sanjay and Alan Bateman I decided to send in my application even though Hadoop hasn't listed any projects on ASFs website. It's up for review at: http://docs.google.com/Doc?docid=dc5ttndv_36d83k5wqx (and, officially, at http://socghop.appspot.com/student_proposal/show/google/gsoc2009/l/t123854188327 for mentors).
Hopefully, you guys think it's a good idea, too. I'll be very glad for any suggestions or ideas. (The application deadline has passed, so the application can't be changed. But all other feedback is appreciated.)
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661647#action_12661647 ]
Doug Cutting commented on HADOOP-4952:
--------------------------------------
> I guess I am missing this library argument [ ... ]
What I mean is that there isn't a clear line between application and library code, that folks might wish to be able to, e.g., start running jobs on two clusters at once, without having to refactor everything to keep track of the config. We should not encourage a style that makes this kind of thing more difficult, but rather one that makes it easy.
> Most file system interfaces are dead simple. I think Hadoop's API can be made equally simple [ ... ]
I agree. But I also think we need to pass the configuration explicitly, and not depend on a static configuration. Our current convention is that only command-line tools create all-new configurations and I think there are good reasons to stick with that.
> In most file systems, the config state is provided by the underlying OS;
The configuration is the equivalent of the unix environment. Using per-process environment variables makes sense when you start new processes for new operations, the unix convention, but that's not the way folks generally work in Java. We could use a static configuration, in fact we used to long ago, but that caused lots of problems. In other projects I've frequently seen folks start out using Java system properties, then switch to something that's passed dynamically. That sort of switch is painful and something we should avoid and encourage others to avoid.
> In my proposal the libray writer will have to add a line "Files.importConfig(configArg)".
That does not work for multi-threaded applications, where different threads are configured differently. Consider, e.g., a Nutch-based system that has different search zones (like, e.g. http://www.archive-it.org/). This is a real and common use case. Some developers use Hadoop's APIs directly, but many if not more layer things on top of it (Hive, HBase, Pig, Jaql, Nutch). Its reasonable for any of these, or just about any Hadoop application, to be repackaged as a multi-user service. Folks should not have to refactor to do that.
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661397#action_12661397 ]
Sanjay Radia commented on HADOOP-4952:
--------------------------------------
Doug> Application code often becomes library code, and changing code that does not explicitly pass a configuration to start passing a configuration breaks compatibility.
I guess I am missing this library argument and having a hard time buying the argument "the rest of the Hodoop APIs do this and so we need to do this here". Most systems do not have such a config parameter in their APIs and they support both Application code and library code. What is so different about Hadoop and its libraries? Some example of the libraries you have in mind would help.
In most file systems, the config state is provided by the underlying OS; it include the root and the working dir. Libraries need to set these if the default is not good enough.
Most file system interfaces are dead simple. I think Hadoop's API can be made equally simple and at the same time allow complex libraries to be built relatively easily. I thought the exportConfig/importConfig facilitates that.
Your use-case is where a piece of application code suddenly turns into library code.
The application code would have used the default config; when it turns into a library, the line of code in your proposal that says "config = new config()" would have to change to "config = CallersConfigArg".
In my proposal the libray writer will have to add a line "Files.importConfig(configArg)".
Either way one line of code has to change: setting the default config or passing in a different config argument.
I think there is more to the underlying environment than just the config; hence libraries have to be more careful . Owen, Arun and I discussed the Task Tracker and whether or not it can do any work on behalf of the job (either a some pre-work (loading cache) or the task itself. The task tracker has to take the credentials and the class path from the job submitter's environment. For example when the task tracker acts on behalf of the job submitter, it has to do the equivalent of JAAS's callerSubject.doAs().
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689045#action_12689045 ]
Doug Cutting commented on HADOOP-4952:
--------------------------------------
> In the NIO.2 API there is no concept of a working directory.
Rather I suspect the working directory concept in NIO.2 is specific to the URI scheme, as in Hadoop, that for file: uri's the java processes working directory would apply, so that file:foo refers to foo in the connected directory in both APIs.
A material difference (that Sanjay alluded to above) is that the default filesystem (uri scheme & authority) in NIO.2 is fixed to be the local filesystem (file:), while in Hadoop it is configurable. So "foo" in NIO.2 always refers to "file:foo", while in Hadoop it might refer to "hdfs:foo" if HDFS is configured as the default filesystem.
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660844#action_12660844 ]
Doug Cutting commented on HADOOP-4952:
--------------------------------------
It might be nice to make it simpler to open a file given a URI and assuming the default configuration, but there are hazards to that as well. Application code often becomes library code, and changing code that does not explicitly pass a configuration to start passing a configuration breaks compatibility. The Hadoop API convention is everything must be explicitly passed a configuration, so that we do not rely on static state. This is just a cost of doing business, and it's dangerous to remove it.
So I agree that static utility methods would be nice, since the FileSystem implementation can be automatically inferred from the configuration and the URI, but I think the Configuration should still be explicitly passed to these methods. If we don't think that applications should ever need to call the non-static FileSystem methods, then placing the static methods in a different class makes sense, since it consolidates the documentation that users need be aware of. Does that make sense to you, Tom?
Also, create() has too many parameters and is fragile. We should add a FileCreation class that encapsulates these. We should consider adding such classes for any methods that take more than a Configuration and a Path.
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676012#action_12676012 ]
Sanjay Radia commented on HADOOP-4952:
--------------------------------------
I have taken an initial look at the new NIO2 comping in jdk7 (JSR 203 - http://openjdk.java.net/projects/nio/)
NIO2 has notions of multiple file system providers (like hadoop) and also has a URI based file system.
Technical considerations for adopting JSR 203:
# Does our file naming fit with that of NIO2?
# Is the NIO2 FileSystem provider interface sufficient for expressing hadoop. ( Hadoop-3518 proposes this)
# Others?
There are non-technical issues such as the timing of NIO2 and jdk7; Alan Bateman,
the spec lead for JSR 203 has informed that jdk7 will come out in 2010 and that google is considering backporting NIO2 to jdk 6 under a different package name.
In NIO2 the default file system (ie slash relative names like /foo/bar) are relative to the JVM's file system.
wd-relative file names are also resolved relative to the JVM's file system.
This makes a lot of sense in Java: Java needs to be consistent with the older Java File system related API; ie a slash-relative and wd-relative are the same whether you use the NIO2 APIs or the other java io APIs.
However, Hadoop has a slightly different notion of default file system. Hadoop file names are as follows
# Fully qualified URIs are resolved against the specified file system: hdfs://nn_adr:port/fo/bar
# file:/// are relative the local file system
# Slash relative names: - these are resolved relative to the default hadoop file system
Typically in a Hadoop cluster the default file system is the file system of the cluster. Application by default inherit this config; a user can set his default file system to any file system.
# wd relative names are relative to the working directory
Pathnames of type (3) or (4) are not consistent with NIO2 and the rest of Java io.
Consider alternate name conventions like hfs:/foo/bar and hfs:foo/bar as Hadoop default root and hadoop working directory: IMHO, these will be too inconvenient: e.g. at the shell, instead of typing "ls foo/bar" one would have to type ls hfs:foo/bar" to list files relative to the working directory.
I don't see how we can escape this dual notion of root and wd:
In the code the distinction using jsr 203 APIs will be roughtly:
{code}
Path p = Paths.get("/foo/bar"); // relative to jvm default fs.
Path p = HDFS_provider.get("/foo/bar"); // relative to hadoop's defautl fs.
Path q = Paths.get("foo/bar"); // relative to jvm's default fs wd.
Path q = HDFS_provider.get("foo/bar"); // relative to hadoop's wd.
{code}
The other thing to evaluate is the file system provider interface. Please continue this discussion on HADOOP-3518. If JSR 203 provider interfaces are not
sufficient then we need to work with the JSR203 expert group to influence its APIs.
Also wrt timing we will have to move forward with our APIs for hadoop 1.0
since 203 will not be finalized by the time Hadoop 1.0 happens.
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663231#action_12663231 ]
Sanjay Radia commented on HADOOP-4952:
--------------------------------------
Thanks for the clarifications; I understand the issue better.
>> In my proposal the libray writer will have to add a line "Files.importConfig(configArg)".
>That does not work for multi-threaded applications, where different threads are configured differently.
Depends on how you implement it. Thread local variables (as we do for the UGI credentials) would address MT..
We have at least the following 3 options
1) Class with static methods (as per patch) using thread local variables to store the
default config.
2) Class with static methods and a config parameter for each method
3)Class with instance methods
{code}
Class Files {
static Files defaultConf();
Files(config)
//open, create etc are all instance methods
}
myFiles = Files.defaultConf(); // common usage
myFiles = new Files(myConf); // for those that want a specific conf
myFiles.create(...)
{code}
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660897#action_12660897 ]
Sanjay Radia commented on HADOOP-4952:
--------------------------------------
Doug do you mean you would prefer something like:
Files.create(config, path, ...)
File.open(config, path, ...)
The exportConfig and importConfig methods in the proposed patch (Owen's suggestion) are for the library writer, but the library writer has to be aware of it.
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660900#action_12660900 ]
Sanjay Radia commented on HADOOP-4952:
--------------------------------------
> Also, create() has too many parameters and is fragile.
The proposed Files interface significantly reduces the number of create methods. It accomplishes this partly by
allowing one to pass in a -1 to certain parameters (blk size, replication factor, buf size) to use their default values
Thus the first create method is very simple and robust for type checking:
{code}
public static FSDataOutputStream create(Path f,
FsPermission permission,
boolean overwrite) throws IOException
{code}
The second create methods is fragile since many of the parameters are ints and one can easily mix the parameters.
Given that most app writers will use the above simpler method, I am not as worried as I used to be.
Were you thinking of something like:
{code}
// Style 1
createdFile = create(path, permission, overwrite)
createFile.setBlockSize(512); // optional most are okay with defaults
outputStream = createFile.open();
// Style 2
outputStream = create(path, permission, overwrite); // the common case where the defaults are fine
createParms.setBlockSize(512); // for when the defaults are not fine.
outputStream = create(path, permission, overwrite, createParms);
// For comparison, the proposed patch would have required the following
outputStream = create(path, permissions, override); // common case when defaults are fine
outputStream = create(path, permissions, override, 512, -1, -1, progress); // when you want to override the defaults
{code}
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4952) Improved files system interface for
the application writer.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660931#action_12660931 ]
Doug Cutting commented on HADOOP-4952:
--------------------------------------
> Doug do you mean you would prefer something like: Files.create(config, path, ...)
Yes. We require a configuration in nearly every other API so I don't think eliminating one here actually saves much and it makes it likely that folks will write code that mistakenly assumes the default configuration.
As for create options we might try something like:
outputStream = File.create(config, path, new FsCreate().permission(p).blocksize(b));
We might also provide a version like:
outputStream = File.create(config, path);
that is equivalent to:
outputStream = File.create(config, path, new FsCreate());
> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
> Key: HADOOP-4952
> URL: https://issues.apache.org/jira/browse/HADOOP-4952
> Project: Hadoop Core
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.
> - Filesystem interface has a confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> - The client side config can be simplified.
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
> -
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.