You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2011/06/02 01:58:48 UTC
[jira] [Created] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Use ServiceLoader to discover compression codec classes
-------------------------------------------------------
Key: HADOOP-7350
URL: https://issues.apache.org/jira/browse/HADOOP-7350
Project: Hadoop Common
Issue Type: Improvement
Components: conf, io
Reporter: Tom White
By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7350:
------------------------------
Attachment: HADOOP-7350.patch
I moved it to take advantage of the ServiceLoader's caching of providers. This new patch does that but moves the iteration back to the getCodecClasses() method. (This is now like the example in the ServiceLoader documentation.)
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207154#comment-13207154 ]
Alejandro Abdelnur commented on HADOOP-7350:
--------------------------------------------
Tom, thanks for reviving this one. Things look OK, the only thing is that I'd revert the logic to first load the codecs defined as services and after that the ones defined in Hadoop's configuration. By doing that, it is possible for a user to override a provided implementation.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237250#comment-13237250 ]
Hadoop QA commented on HADOOP-7350:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12514395/HADOOP-7350.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/768//console
This message is automatically generated.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257741#comment-13257741 ]
Hudson commented on HADOOP-7350:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2125 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2125/])
HADOOP-7350. Use ServiceLoader to discover compression codec classes. (Revision 1328083)
Result = ABORTED
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328083
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.io.compress.CompressionCodec
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodecFactory.java
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.0
>
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7350:
------------------------------
Attachment: HADOOP-7350.patch
Sorry - this should be the right patch now.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7350:
------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Target Version/s: 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I fixed the nit and committed this. Thanks for the review Alejandro.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.0
>
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045127#comment-13045127 ]
Todd Lipcon commented on HADOOP-7350:
-------------------------------------
hmm, the two most recent patches are identical best I can see...
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257710#comment-13257710 ]
Hudson commented on HADOOP-7350:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #2108 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2108/])
HADOOP-7350. Use ServiceLoader to discover compression codec classes. (Revision 1328083)
Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328083
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.io.compress.CompressionCodec
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodecFactory.java
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045216#comment-13045216 ]
Todd Lipcon commented on HADOOP-7350:
-------------------------------------
Curious: why the change to only load once in the static initializer?
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7350:
------------------------------
Attachment: HADOOP-7350.patch
Yes, this got forgotten. I've updated the patch to the new source structure, and added a couple of tests to check that Snappy and LZ4 extensions are picked up.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7350:
------------------------------
Attachment: HADOOP-7350.patch
Slight adjustment to load codec classes only once using a ServiceLoader.
I'll address the HDFS documentation change in another JIRA.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7350:
------------------------------
Attachment: HADOOP-7350.patch
Updated patch to address Alejandro's feedback. Note that codecs defined in the configuration still take precedence over those picked up by the service loader.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045564#comment-13045564 ]
Hadoop QA commented on HADOOP-7350:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481726/HADOOP-7350.patch
against trunk revision 1132776.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/587//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/587//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/587//console
This message is automatically generated.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7350:
------------------------------
Attachment: HADOOP-7350.patch
This patch modifies CompressionCodecFactory.getCodecClasses() to use a service loader in addition to reading class names from io.compression.codecs.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Attachments: HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7350:
------------------------------
Assignee: Tom White
Status: Patch Available (was: Open)
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Owen O'Malley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-7350:
----------------------------------
Status: Open (was: Patch Available)
It needs to be rebased to the mavenized source tree.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207536#comment-13207536 ]
Alejandro Abdelnur commented on HADOOP-7350:
--------------------------------------------
I may have expressed incorrectly, my suggestion was that in the getCodecClasses(Configuration) method, the loop loading the codecs from CODEC_PROVIDERS should be done before loading the codecs from the Hadoop 'io.compression.codec' property. Also, in the CompressionCodecFactory() constructor, doing a reverse of the list seems wrong as that would effectively put the service loaded ones first. I think that the reverse should not be there, then the list of codecs is ALL_SERVICE_CODECS (in non-deterministic order) and then the configuration set ones, in order or appearance where the last one overrides.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192459#comment-13192459 ]
Hadoop QA commented on HADOOP-7350:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511199/HADOOP-7350.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 7 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/528//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/528//console
This message is automatically generated.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated HADOOP-7350:
----------------------------------------
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.2)
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043067#comment-13043067 ]
Todd Lipcon commented on HADOOP-7350:
-------------------------------------
+1 on most recent patch
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7350:
------------------------------
Target Version/s: 0.23.1
Status: Patch Available (was: Open)
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043063#comment-13043063 ]
Hadoop QA commented on HADOOP-7350:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481287/HADOOP-7350.patch
against trunk revision 1130758.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/563//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/563//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/563//console
This message is automatically generated.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042554#comment-13042554 ]
Todd Lipcon commented on HADOOP-7350:
-------------------------------------
- We should probably remove the codecs from core-default.xml now that they're loaded via ServiceLoader
- Is there a way to inject a new codec programatically through the ServiceLoader interface? If so, we could entirely deprecate io.compression.codecs. If not, maybe we should rename it to something like io.compression.extra.codecs and specify that it's only necessary if you have a codec that doesn't expose itself through ServiceLoader?
- hdfs-default.xml has an item dfs.image.compression.codec that needs to be updated
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043066#comment-13043066 ]
Todd Lipcon commented on HADOOP-7350:
-------------------------------------
bq. This doesn't need to be updated
Sorry, the default doesn't need to change, but the docs should be updated: it no longer needs to be one of the codecs listed in io.compression.codecs. It just needs to be any codec that's "registered" (either via conf or via ServiceLoader). I don't know the best verbiage to succinctly document it, though.
The rest of your points make sense.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257590#comment-13257590 ]
Alejandro Abdelnur commented on HADOOP-7350:
--------------------------------------------
+1
small NIT:
Where you do:
{code}
+ List<Class<? extends CompressionCodec>> result
+ = new ArrayList<Class<? extends CompressionCodec>>();
+ // Add codec classes discovered via service loading
+ for (CompressionCodec codec : CODEC_PROVIDERS) {
+ if (!result.contains(codec.getClass())) {
+ result.add(codec.getClass());
+ }
+ }
{code}
The IF NOT is not required as the ServiceLoader will ignore duplicate classes in a non-deterministic way per ServiceLoader javadocs.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258206#comment-13258206 ]
Hudson commented on HADOOP-7350:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #1020 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1020/])
HADOOP-7350. Use ServiceLoader to discover compression codec classes. (Revision 1328083)
Result = FAILURE
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328083
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.io.compress.CompressionCodec
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodecFactory.java
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.0
>
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043143#comment-13043143 ]
Hadoop QA commented on HADOOP-7350:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481304/HADOOP-7350.patch
against trunk revision 1130833.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/565//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/565//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/565//console
This message is automatically generated.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7350:
------------------------------
Attachment: HADOOP-7350.patch
Thanks for taking a look, Alejandro. Here's an updated patch in which codecs defined in the configuration take precedence over those picked up by the service loader.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253653#comment-13253653 ]
Hadoop QA commented on HADOOP-7350:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522601/HADOOP-7350.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/850//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/850//console
This message is automatically generated.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042528#comment-13042528 ]
Hadoop QA commented on HADOOP-7350:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481172/HADOOP-7350.patch
against trunk revision 1129989.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/557//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/557//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/557//console
This message is automatically generated.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7350:
------------------------------
Attachment: HADOOP-7350.patch
{quote}
- We should probably remove the codecs from core-default.xml now that they're loaded via ServiceLoader
{quote}
Done - see new patch.
{quote}
- Is there a way to inject a new codec programatically through the ServiceLoader interface? If so, we could entirely deprecate io.compression.codecs. If not, maybe we should rename it to something like io.compression.extra.codecs and specify that it's only necessary if you have a codec that doesn't expose itself through ServiceLoader?
{quote}
I don't think we need to deprecate or rename io.compression.codecs - it's just used to specify _additional_ codecs to the ones that are loaded through a ServiceLoader. Note that duplicates are ignored, so there's no problem with users older configs having codecs that could be loaded through ServiceLoader.
{quote}
- hdfs-default.xml has an item dfs.image.compression.codec that needs to be updated
{quote}
This doesn't need to be updated, although with HADOOP-7323 (and a corresponding HDFS change) it could be changed to "default".
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045155#comment-13045155 ]
Hadoop QA commented on HADOOP-7350:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481630/HADOOP-7350.patch
against trunk revision 1132776.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/586//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/586//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/586//console
This message is automatically generated.
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Owen O'Malley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185686#comment-13185686 ]
Owen O'Malley commented on HADOOP-7350:
---------------------------------------
Tom,
We should apply this. Can you rebase it to reflect the mavenization?
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257709#comment-13257709 ]
Hudson commented on HADOOP-7350:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2182 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2182/])
HADOOP-7350. Use ServiceLoader to discover compression codec classes. (Revision 1328083)
Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328083
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.io.compress.CompressionCodec
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodecFactory.java
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
compression codec classes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258227#comment-13258227 ]
Hudson commented on HADOOP-7350:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1055 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1055/])
HADOOP-7350. Use ServiceLoader to discover compression codec classes. (Revision 1328083)
Result = FAILURE
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1328083
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.io.compress.CompressionCodec
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodecFactory.java
> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
> Key: HADOOP-7350
> URL: https://issues.apache.org/jira/browse/HADOOP-7350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf, io
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.0
>
> Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira