You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Alejandro Abdelnur (JIRA)" <ji...@apache.org> on 2011/08/17 21:31:27 UTC
[jira] [Created] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Use JDK ServiceLoader mechanism to find FileSystem implementations
------------------------------------------------------------------
Key: HADOOP-7549
URL: https://issues.apache.org/jira/browse/HADOOP-7549
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 0.23.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 0.23.0
Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086538#comment-13086538 ]
Hadoop QA commented on HADOOP-7549:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12490690/HADOOP-7549v1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/46//console
This message is automatically generated.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Tsz Wo (Nicholas), SZE (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208120#comment-13208120 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-7549:
------------------------------------------------
> test-patch fails with patches spanning across projects.
This becomes a problem: patches spanning across projects usually are more involved but we run less tests on it. Is there a way to fix it?
For the moment, we should manually check for the findbugs, javac and javadoc warnings, and run the unit tests before committing patches.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13095434#comment-13095434 ]
Alejandro Abdelnur commented on HADOOP-7549:
--------------------------------------------
@Owen, are you suggesting we should de-overlap schema names? If so, agreed.
Some how related, it was mentioned before, the patch allows you to override via configuration settings an impl defined as service. This will allow people to use alternate impls for a FS schema easily.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087422#comment-13087422 ]
Suresh Srinivas commented on HADOOP-7549:
-----------------------------------------
> From a user perpective, just adding a JAR with a new FileSystem is enough to use it, no need to set configurations in conf/ or in their APP.
Typically configuration is preset in core-default.xml and users do not need to change it.
Only frameworks such as mapreduce and raid change it.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated HADOOP-7549:
---------------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.24.0)
2.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
committed to trunk and branch-2. (committed v2 as v3 which is identical was missing the service files)
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207950#comment-13207950 ]
Alejandro Abdelnur commented on HADOOP-7549:
--------------------------------------------
test-patch fails with patches spanning across projects.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Suresh Srinivas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207978#comment-13207978 ]
Suresh Srinivas commented on HADOOP-7549:
-----------------------------------------
Alejandro, how does it handle the following:
bq. multiple implementations could be using the same scheme names (example RawLocalFileSystem, LocalFileSystem etc). I am not sure which one wins in that case. Also some one adding a jar to the path, with say an experimental file system (say HDFS) could override the DistributedFileSystem and not even know about it.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261626#comment-13261626 ]
Hudson commented on HADOOP-7549:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1060 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1060/])
HADOOP-7549. Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087424#comment-13087424 ]
Alejandro Abdelnur commented on HADOOP-7549:
--------------------------------------------
not really, If you are using a custom FS implementation (ie HOOP) currently it has to be added in the client side.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087953#comment-13087953 ]
Alejandro Abdelnur commented on HADOOP-7549:
--------------------------------------------
Regarding the first concern, conf wins over service. So there is always an override.
Regarding the second concern, you are right in a sense. That is similar to adding a JAR with duplicated classes.
Adding Hoop to the config file would be a point solution for Hoop, but not for other FS impls.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086536#comment-13086536 ]
Alejandro Abdelnur commented on HADOOP-7549:
--------------------------------------------
The current loading mechanism (from Hadoop configuration files) would be preserved and have precedence (although it would be deprecated).
A new method {{getScheme()}} in the {{FileSystem}} class would publish the scheme handled by the implementation.
The base {{FileSystem}} class will implement this method throwing an {{UnsupportedOperationException}}.
All {{FileSystem}} implementations in Hadoop will be retrofitted to implement this method declaring their scheme and they will be registered as services.
Custom {{FileSystem}} implementations not part of Hadoop will not break if they don't implement the {{getScheme()}} method. The will continue to work as before, (they just won't support the service loading). In a future release they could be retrofitted to use the service mechanism (if their authors choose to).
Because implementations registered in the configuration have precedence over implementations registered as services, modified implementations of Hadoop provided implementations will still override the service definitions.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208104#comment-13208104 ]
Alejandro Abdelnur commented on HADOOP-7549:
--------------------------------------------
@Suresh, If both impls using the same scheme are defined as services, then it is undetermined (by JDK specs). If one is defined as service an the other in the configuration, the one in the configuration wins. If both are defined in the configuration, the latter one wins.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated HADOOP-7549:
---------------------------------------
Attachment: HADOOP-7549v2.patch
Patch rebased to trunk.
It keeps current mappings in the services definition (it was been done already).
Definition in configuration overrides a service definition (it was been done already).
It removes the deprecated warning log when resolving a FS implementation from configuration as this is the way of overriding an implementation, thus it is not deprecated.
It adds HAR and WebHdfs filesystems to the service loading mechanism/definitions.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13095497#comment-13095497 ]
Suresh Srinivas commented on HADOOP-7549:
-----------------------------------------
bq. "file" scheme to point to RawLocalFileSystem, "checksum" scheme to point to LocalFileSystem
This is backward incompatible. We should retain the definitions that are in core-default.xml for mapping scheme to a file system. By that definition, scheme "file" should map to LocalFileSystem. I am fine moving current schemes from configuration file to ServiceLoader based mechanism.
We should also retain the configuration based mechanism. This as I indicated earlier, gives flexibility during run time to override a scheme to a different file system mapping.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated HADOOP-7549:
---------------------------------------
Attachment: HADOOP-7549v3.patch
reattaching with correct patch filename
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087854#comment-13087854 ]
Suresh Srinivas commented on HADOOP-7549:
-----------------------------------------
My main concern is - multiple implementations could be using the same scheme names (example RawLocalFileSystem, LocalFileSystem etc). I am not sure which one wins in that case. Also some one adding a jar to the path, with say an experimental file system (say HDFS) could override the DistributedFileSystem and not even know about it.
We should consider adding HOOP to the config file.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257846#comment-13257846 ]
Hadoop QA commented on HADOOP-7549:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523408/HADOOP-7549v3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/869//console
This message is automatically generated.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087158#comment-13087158 ]
Suresh Srinivas commented on HADOOP-7549:
-----------------------------------------
How does this handle two file system implementations with same scheme? Currently it is handled by picking the one in the configuration. Also, there might be code in mapreduce, that could be making use of config based mechanism for choosing file systems.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated HADOOP-7549:
---------------------------------------
Attachment: HADOOP-7549v1.patch
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260995#comment-13260995 ]
Hudson commented on HADOOP-7549:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2200 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2200/])
HADOOP-7549. Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257850#comment-13257850 ]
Alejandro Abdelnur commented on HADOOP-7549:
--------------------------------------------
test-patch failure due to cross module patch contents.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Jane Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485135#comment-13485135 ]
Jane Chen commented on HADOOP-7549:
-----------------------------------
Hi Alejandro,
I was calling FileSystem.get(URI, conf, user) from C++ code through JNI. After upgrading to 2.0, this call is giving me "java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.LocalFileSystem not found" error. It doesn't seem to be a classpath issue, since I can instantiate the class with no problem. I'd like to know whether you have any insight on this.
Also I read in the bug that there's a configuration to disable loading through ServiceLoader. I'd like to try that, and would appreciate if you could let me know what's the proper way to do that.
Thanks,
Jane
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0-alpha
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087395#comment-13087395 ]
Alejandro Abdelnur commented on HADOOP-7549:
--------------------------------------------
>From a user perpective, just adding a JAR with a new FileSystem is enough to use it, no need to set configurations in conf/ or in their APP.
Regarding the direct use use of 'fs.*.impl', I want to take a look to see why is that needed.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260999#comment-13260999 ]
Hudson commented on HADOOP-7549:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #2126 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2126/])
HADOOP-7549. Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated HADOOP-7549:
---------------------------------------
Attachment: HDFS-7549v3.patch
rebasing to trunk to remove patch complains.
I've run test-patch locally with and without the patch and I get the same number of warnings:
{code}
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
-1 javadoc. The javadoc tool appears to have generated 8 warning messages.
-1 javac. The applied patch generated 1707 javac compiler warnings (more than the trunk's current 1706 warnings).
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 20 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated HADOOP-7549:
---------------------------------------
Attachment: (was: HDFS-7549v3.patch)
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260862#comment-13260862 ]
Tom White commented on HADOOP-7549:
-----------------------------------
+1 This looks good to me. Existing configurations will still work with this patch, so there are no compatibility concerns.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13095483#comment-13095483 ]
Owen O'Malley commented on HADOOP-7549:
---------------------------------------
Sorry, I wasn't clear. Yes, part of using ServiceLoader to load these impls is that we need to make the schemes for each FileSystem unique. I was proposing that we change
"file" scheme to point to RawLocalFileSystem
"checksum" scheme to point to LocalFileSystem
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087258#comment-13087258 ]
Suresh Srinivas commented on HADOOP-7549:
-----------------------------------------
bq. Maybe we shouldn't say config loading is deprecated, and we should remove the warning I'm putting there.
That is what I was getting at
bq. Regarding mapreduce, would you point me to the place where that happens?
grep fs.*impl in java files
Given that we cannot deprecate the config based mechanism, which in addition to the mechanism you are proposing, allows changing FileSystem implementation during run time, why do we need this scheme?
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261009#comment-13261009 ]
Hudson commented on HADOOP-7549:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2142 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2142/])
HADOOP-7549. Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Jane Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485314#comment-13485314 ]
Jane Chen commented on HADOOP-7549:
-----------------------------------
Hi Alejandro,
Thank you for your input.
I was using the bundled jars directly. I found that I have to call FileSystem.loadFileSystems() in the same thread where JVM is initialized. So I got it working now.
I'm guessing the class loader used in the ServiceLoader doesn't get the classpath set if it isn't the same thread creating the JVM. But I haven't tried to prove it. I'd love to hear it if you have any insight on that.
Thanks again,
Jane
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0-alpha
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207928#comment-13207928 ]
Hadoop QA commented on HADOOP-7549:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12514517/HADOOP-7549v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/596//console
This message is automatically generated.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262245#comment-13262245 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-7549:
------------------------------------------------
It seems that this breaks FsUrlStreamHandlerFactory; see also HDFS-3323.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485305#comment-13485305 ]
Alejandro Abdelnur commented on HADOOP-7549:
--------------------------------------------
Jane,
If you are unjaring/rejaring Hadoop JARs when using Hadoop from a JVM started from C++ then that could be the issue. JDK ServiceLoader uses the same file name in multiple JARs to pick up all services. If you unjar/rejar the JARs, you are looking all but one of those files.
You could try using the old mechanism for registering filesystems in the hadoop configuration files 'fs.<SCHEME>.impl=<CLASS>.
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0-alpha
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13095418#comment-13095418 ]
Owen O'Malley commented on HADOOP-7549:
---------------------------------------
I think we need to move this direction. The cases where we have multiple file systems with the same "protocol" are very problematic, especially as we move to the FileContext API.
We should probably rename:
file -> raw local
checksum -> local
are there any other overlapping pairs?
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087169#comment-13087169 ]
Alejandro Abdelnur commented on HADOOP-7549:
--------------------------------------------
Suresh,
Configuration defined filesystems have priority over service defined file systems.
Maybe we shouldn't say config loading is deprecated, and we should remove the warning I'm putting there.
Regarding mapreduce, would you point me to the place where that happens?
Thanks.
Alejandro
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261612#comment-13261612 ]
Hudson commented on HADOOP-7549:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #1025 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1025/])
HADOOP-7549. Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated HADOOP-7549:
---------------------------------------
Status: Patch Available (was: Open)
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HADOOP-7549v1.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira