You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Arvid Heise (Jira)" <ji...@apache.org> on 2019/12/12 13:33:00 UTC
[jira] [Commented] (FLINK-15215) Not able to provide a custom AWS
credentials provider with flink-s3-fs-hadoop
[ https://issues.apache.org/jira/browse/FLINK-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994667#comment-16994667 ]
Arvid Heise commented on FLINK-15215:
-------------------------------------
For 1.10, we want to get rid of shading altogether. We fixed one bug FLINK-14574 that prevented us from doing so and we are in the process to remove the shading in [https://github.com/apache/flink/pull/10482] . The release is planned in the beginning of January.
We are also backporting the fix to 1.9.0, but it's unlikely that we will also backport the shading as it may break configuration. However, once 1.9.2 is out, you could probably use the 1.10.0 s3 libraries as plugins.
Nevertheless, 1.9.2 will probably not appear much sooner and since you run on EMR, it will also take some time until you could use that. Once we merge the PR, you could of course try out the unshaded snapshot versions of 1.10. If you are not hitting the above mentioned bug (only affect batch outputs and not streaming), it could already work for you.
> Not able to provide a custom AWS credentials provider with flink-s3-fs-hadoop
> -----------------------------------------------------------------------------
>
> Key: FLINK-15215
> URL: https://issues.apache.org/jira/browse/FLINK-15215
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Affects Versions: 1.9.0
> Reporter: Tank
> Priority: Major
>
> I am using Flink 1.9.0 on EMR (emr 5.28), with StreamingFileSink using an S3 filesystem.
> I want flink to write to an S3 bucket which is running in another AWS account, and I want to do that by assuming a role in the other account. This can be easily accomplished by providing a custom credential provider (similar to [https://aws.amazon.com/blogs/big-data/securely-analyze-data-from-another-aws-account-with-emrfs/])
>
> As described in https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/#pluggable-file-systems, I copied {{flink-s3-fs-hadoop-1.9.0.jar}} to the plugins directory. But the configuration parameter 'fs.s3a.aws.credentials.provider' is getting shaded [https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java#L47]}}, and so are all the aws sdk dependencies, so when I provide a custom credential provider, it complained that I was not implementing the correct interface (AWSCredentialsProvider)
> The fix made in https://issues.apache.org/jira/browse/FLINK-13044 allows users to use one of the built-in credential providers like `_InstanceProfileCredentialsProvider`,_ but still does not help with providing custom credential providers.
>
> Related: https://issues.apache.org/jira/browse/FLINK-13602
--
This message was sent by Atlassian Jira
(v8.3.4#803005)