You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Chesnay Schepler (JIRA)" <ji...@apache.org> on 2019/08/06 19:45:00 UTC
[jira] [Closed] (FLINK-13044) Shading of AWS SDK in
flink-s3-fs-hadoop results in ClassNotFoundExceptions
[ https://issues.apache.org/jira/browse/FLINK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chesnay Schepler closed FLINK-13044.
------------------------------------
Resolution: Fixed
master: 5e5f5bec72d30397253fc1f11105ca385ca8681f
1.9: 16f519a1e7efae9210725f741c27a647ac3f3271
> Shading of AWS SDK in flink-s3-fs-hadoop results in ClassNotFoundExceptions
> ---------------------------------------------------------------------------
>
> Key: FLINK-13044
> URL: https://issues.apache.org/jira/browse/FLINK-13044
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Affects Versions: 1.8.0
> Reporter: Sebastian J.
> Assignee: Chesnay Schepler
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The current build configuration shades the AWS SDK into {{flink-s3-fs-hadoop.jar}}
> Due to [MSHAD-156|https://issues.apache.org/jira/browse/MSHADE-156] the shading itself does not only add the classes to the JAR but also alters the string literal {{PACKAGE_PREFIXES_TO_SHADE}} in {{org.apache.flink.fs.s3hadoop.S3FileSystemFactory}} from {{com.amazonaws.}} to {{org.apache.flink.fs.s3base.shaded.com.amazonaws}} - which causes settings like
> {noformat}fs.s3a.aws.credentials.provider: com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
> to not be remapped properly as the {{startsWith}} in {{org.apache.flink.fs.s3.common.HadoopConfigLoader.shadeClassConfig()}} doesn't match the beginning of the FQCN.
> This is a known issue in the Maven shade plugin (see [MSHAD-156|https://issues.apache.org/jira/browse/MSHADE-156]) - a possible workaround is named in the ticket, too.
> This issue is only visible looking into the compiled {{class}} file after the shading happened!
> Fixing the {{PACKAGE_PREFIXES_TO_SHADE}} value alone doesn't help as based on {{FLINK_SHADING_PREFIX}} the FQCN is assembled as {noformat}org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat} which can't be found either;
> the class is located at {noformat}org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)