[jira] [Commented] (FLINK-13044) Shading of AWS SDK in flink-s3-fs-hadoop results in ClassNotFoundExceptions

["Stephan Ewen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882139#comment-16882139 ] 

Stephan Ewen commented on FLINK-13044:
--------------------------------------

We can in theory drop shading of the filesystems all together now, and instead put them in the plugins folder of Flink.
That also gives class loader isolation and avoid class path pollution, and does not require an shading.

Because this is very new, we did not want to drop shading instantly, in case users still put the file system into "/lib".

As a workaround, you could comment out the class relocation part of the build, and put the resulting artifacts into "/plugins".
That should do the trick.

I hope that we will drop the evil shading magic completely from 1.10.

> Shading of AWS SDK in flink-s3-fs-hadoop results in ClassNotFoundExceptions
> ---------------------------------------------------------------------------
>
>                 Key: FLINK-13044
>                 URL: https://issues.apache.org/jira/browse/FLINK-13044
>             Project: Flink
>          Issue Type: Bug
>          Components: BuildSystem / Shaded, Connectors / Hadoop Compatibility
>    Affects Versions: 1.8.0
>            Reporter: Sebastian J.
>            Priority: Critical
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
>  The current build configuration shades the AWS SDK into {{flink-s3-fs-hadoop.jar}}
> Due to [MSHAD-156|https://issues.apache.org/jira/browse/MSHADE-156] the shading itself does not only add the classes to the JAR but also alters the string literal {{PACKAGE_PREFIXES_TO_SHADE}} in {{org.apache.flink.fs.s3hadoop.S3FileSystemFactory}} from {{com.amazonaws.}} to {{org.apache.flink.fs.s3base.shaded.com.amazonaws}} - which causes settings like 
> {noformat}fs.s3a.aws.credentials.provider: com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat}
> to not be remapped properly as the {{startsWith}} in {{org.apache.flink.fs.s3.common.HadoopConfigLoader.shadeClassConfig()}} doesn't match the beginning of the FQCN.
> This is a known issue in the Maven shade plugin (see [MSHAD-156|https://issues.apache.org/jira/browse/MSHADE-156]) - a possible workaround is named in the ticket, too.
> This issue is only visible looking into the compiled {{class}} file after the shading happened!
> Fixing the {{PACKAGE_PREFIXES_TO_SHADE}} value alone doesn't help as based on {{FLINK_SHADING_PREFIX}} the FQCN is assembled as {noformat}org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat} which can't be found either;
> the class is located at {noformat}org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain{noformat} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)