You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Till Rohrmann (JIRA)" <ji...@apache.org> on 2019/07/15 08:28:00 UTC
[jira] [Assigned] (FLINK-13044) Shading of AWS SDK in
flink-s3-fs-hadoop results in ClassNotFoundExceptions
[ https://issues.apache.org/jira/browse/FLINK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Till Rohrmann reassigned FLINK-13044:
-------------------------------------
Assignee: Chesnay Schepler
> Shading of AWS SDK in flink-s3-fs-hadoop results in ClassNotFoundExceptions
> ---------------------------------------------------------------------------
>
> Key: FLINK-13044
> URL: https://issues.apache.org/jira/browse/FLINK-13044
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Affects Versions: 1.8.0
> Reporter: Sebastian J.
> Assignee: Chesnay Schepler
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The current build configuration shades the AWS SDK into {{flink-s3-fs-hadoop.jar}}
> Due to [MSHAD-156|https://issues.apache.org/jira/browse/MSHADE-156] the shading itself does not only add the classes to the JAR but also alters the string literal {{PACKAGE_PREFIXES_TO_SHADE}} in {{org.apache.flink.fs.s3hadoop.S3FileSystemFactory}} from {{com.amazonaws.}} to {{org.apache.flink.fs.s3base.shaded.com.amazonaws}} - which causes settings like
> {noformat}fs.s3a.aws.credentials.provider: com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
> to not be remapped properly as the {{startsWith}} in {{org.apache.flink.fs.s3.common.HadoopConfigLoader.shadeClassConfig()}} doesn't match the beginning of the FQCN.
> This is a known issue in the Maven shade plugin (see [MSHAD-156|https://issues.apache.org/jira/browse/MSHADE-156]) - a possible workaround is named in the ticket, too.
> This issue is only visible looking into the compiled {{class}} file after the shading happened!
> Fixing the {{PACKAGE_PREFIXES_TO_SHADE}} value alone doesn't help as based on {{FLINK_SHADING_PREFIX}} the FQCN is assembled as {noformat}org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat} which can't be found either;
> the class is located at {noformat}org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)