[jira] [Created] (HDFS-15663) NetworkBinding has a runtime class dependency on a third-party shaded class

["Chris Wensel (Jira)" <ji...@apache.org>]

Chris Wensel created HDFS-15663:
-----------------------------------

             Summary: NetworkBinding has a runtime class dependency on a third-party shaded class
                 Key: HDFS-15663
                 URL: https://issues.apache.org/jira/browse/HDFS-15663
             Project: Hadoop HDFS
          Issue Type: Bug
    Affects Versions: 3.3.0
            Reporter: Chris Wensel


The hadoop-aws library has a dependency on 'com.amazonaws':aws-java-sdk-bundle' which in turn is a fat jar of all AWS SDK libraries and shaded dependencies.

 

This dependency is 181MB.

 

Some applications using the S3AFilesystem may be sensitive to having a large footprint. For example, building an application using Parquet and bundled with Docker.

 

Typically, in prior Hadoop versions, the bundle was replaced by the specific AWS SDK dependencies, dropping the overall footprint.

 

In 3.3 (and maybe prior versions) this strategy does not work because of the following exception: 

{{java.lang.NoClassDefFoundError: com/amazonaws/thirdparty/apache/http/conn/socket/ConnectionSocketFactory}}
{{ at org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1335)}}
{{ at org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1290)}}
{{ at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1247)}}
{{ at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:61)}}
{{ at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:644)}}
{{ at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:390)}}
{{ at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)}}
{{ at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)}}
{{ at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)}}
{{ at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)}}
{{ at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)}}

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org