You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/02/03 11:59:00 UTC
[jira] [Work logged] (HADOOP-17337) NetworkBinding has a runtime
class dependency on a third-party shaded class
[ https://issues.apache.org/jira/browse/HADOOP-17337?focusedWorklogId=546975&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-546975 ]
ASF GitHub Bot logged work on HADOOP-17337:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 03/Feb/21 11:58
Start Date: 03/Feb/21 11:58
Worklog Time Spent: 10m
Work Description: mukund-thakur commented on pull request #2599:
URL: https://github.com/apache/hadoop/pull/2599#issuecomment-772455150
LGTM +1.
I understand that it is not nearly impossible to write a test.
Also I ran the IT suite with my bucket. All good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 546975)
Time Spent: 1h 40m (was: 1.5h)
> NetworkBinding has a runtime class dependency on a third-party shaded class
> ---------------------------------------------------------------------------
>
> Key: HADOOP-17337
> URL: https://issues.apache.org/jira/browse/HADOOP-17337
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Chris Wensel
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The hadoop-aws library has a dependency on 'com.amazonaws':aws-java-sdk-bundle' which in turn is a fat jar of all AWS SDK libraries and shaded dependencies.
>
> This dependency is 181MB.
>
> Some applications using the S3AFilesystem may be sensitive to having a large footprint. For example, building an application using Parquet and bundled with Docker.
>
> Typically, in prior Hadoop versions, the bundle was replaced by the specific AWS SDK dependencies, dropping the overall footprint.
>
> In 3.3 (and maybe prior versions) this strategy does not work because of the following exception:
> {{java.lang.NoClassDefFoundError: com/amazonaws/thirdparty/apache/http/conn/socket/ConnectionSocketFactory}}
> {{ at org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1335)}}
> {{ at org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1290)}}
> {{ at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1247)}}
> {{ at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:61)}}
> {{ at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:644)}}
> {{ at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:390)}}
> {{ at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)}}
> {{ at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)}}
> {{ at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)}}
> {{ at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)}}
> {{ at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org