You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Liu Shaohui (JIRA)" <ji...@apache.org> on 2016/08/11 03:33:20 UTC
[jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to
find FileSystem implementations
[ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15416467#comment-15416467 ]
Liu Shaohui commented on HADOOP-7549:
-------------------------------------
[~jxchen] [~tucu00]
I encountered same problem when calling CompressionCodecFactory from C++ code through JNI.
{code}
java.util.ServiceConfigurationError: org.apache.hadoop.io.compress.CompressionCodec: Provider org.apache.hadoop.io.compress.BZip2Codec not found
at java.util.ServiceLoader.fail(ServiceLoader.java:231)
at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:116)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
{code}
The reason is that calling vm->AttachCurrentThread() in jni_helper.c of libhdfs sets the context class loader to the “bootstrap loader”, which is insufficient for ServiceLoader to function properly.
See:
https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/invocation.html#AttachCurrentThread.
In current implements of libhdfs, the first calling getJNIEnv will create a java vm which doesn't change the context class loader. But later threads calling getJNIEnv will attach thread to the java vm which will change the context class loader. So It works when call FileSystem.loadFileSystems() in the same thread where JVM is initialized.
A suggest from John Lilley is to setting the system class loader to be the thread's context class loader:
{code}
java.lang.Thread.currentThread().setContextClassLoader(java.lang.ClassLoader.getSystemClassLoader());
{code}
ps: You must translate it to the JNI callings.
See : [Mail link | http://mail-archives.apache.org/mod_mbox/hadoop-user/201409.mbox/%3Cfc3d75136d434956ae9cc926213576f2@DM2PR0701MB729.namprd07.prod.outlook.com%3E]
> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>
> Key: HADOOP-7549
> URL: https://issues.apache.org/jira/browse/HADOOP-7549
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.0-alpha
>
> Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
>
>
> Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.
> This is similar to what is being proposed for compression codecs (HADOOP-7350).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org