[jira] [Created] (HADOOP-20034) WebHdfsFileSystem fails with the 3.5.0 shaded client: hadoop-client-runtime has no JAX-RS implementation

["Dan Ziemba (Jira)" <ji...@apache.org>]

Dan Ziemba created HADOOP-20034:
-----------------------------------

             Summary: WebHdfsFileSystem fails with the 3.5.0 shaded client: hadoop-client-runtime has no JAX-RS implementation
                 Key: HADOOP-20034
                 URL: https://issues.apache.org/jira/browse/HADOOP-20034
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 3.5.0
            Reporter: Dan Ziemba


With hadoop-client-api and hadoop-client-runtime 3.5.0, every WebHDFS call that parses a JSON response (getFileStatus, listStatus, globStatus, delete, ...) fails with a ClassNotFoundException for the relocated Jersey RuntimeDelegateImpl. open() still works because it parses no JSON. The same client code works with 3.4.x.

WebHdfsFileSystem.jsonParse() calls javax.ws.rs.core.MediaType.valueOf(contentType), which looks up a JAX-RS RuntimeDelegate. Since HADOOP-19652, hadoop-client-runtime bundles the relocated JAX-RS API, but it still excludes org.glassfish.jersey.core:*, so the default delegate org.apache.hadoop.shaded.org.glassfish.jersey.internal.RuntimeDelegateImpl is missing. In 3.4.x the runtime jar bundled Jersey 1's RuntimeDelegateImpl, which the Jersey 2 upgrade (HADOOP-15984) removed. branch-3.5 still has the same exclusion and call.

*To reproduce* (Java 21; classpath: hadoop-client-api-3.5.0.jar, hadoop-client-runtime-3.5.0.jar, slf4j-api, commons-logging; any WebHDFS or HttpFS endpoint, seen against a 3.4.1 HttpFS):
{code:java}
FileSystem fs = FileSystem.get(URI.create("webhdfs://namenode.example.com:9870"), new Configuration());
fs.getFileStatus(new Path("/"));
{code}
With the 3.4.3 jars, the same code returns the file status.

{noformat}
Exception in thread "main" java.io.IOException: namenode.example.com:9870: Response decoding failure: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.org.glassfish.jersey.internal.RuntimeDelegateImpl
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathResponseRunner.getResponse(WebHdfsFileSystem.java:971)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:829)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:689)
	at java.base/javax.security.auth.Subject.lambda$callAs$0(Subject.java:379)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
	at java.base/javax.security.auth.Subject.callAs(Subject.java:381)
	at org.apache.hadoop.security.authentication.util.SubjectUtil.callAs(SubjectUtil.java:242)
	at org.apache.hadoop.security.authentication.util.SubjectUtil.doAs(SubjectUtil.java:314)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1958)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:685)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:1109)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:1120)
	at Repro.main(Repro.java:9)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.org.glassfish.jersey.internal.RuntimeDelegateImpl
	at org.apache.hadoop.shaded.javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:129)
	at org.apache.hadoop.shaded.javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:96)
	at org.apache.hadoop.shaded.javax.ws.rs.core.MediaType.valueOf(MediaType.java:172)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:496)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathResponseRunner.getResponse(WebHdfsFileSystem.java:961)
	... 13 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.org.glassfish.jersey.internal.RuntimeDelegateImpl
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:423)
	at java.base/java.lang.Class.forName(Class.java:414)
	at org.apache.hadoop.shaded.javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:87)
	at org.apache.hadoop.shaded.javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:185)
	at org.apache.hadoop.shaded.javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:111)
	... 17 more
{noformat}

*Possible fixes:* bundle and relocate jersey-common (which has RuntimeDelegateImpl) in hadoop-client-runtime, or have jsonParse() check the Content-Type without JAX-RS.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org