[jira] [Created] (FLINK-8439) Document using a custom AWS Credentials Provider with flink-3s-fs-hadoop

["Dyana Rose (JIRA)" <ji...@apache.org>]

Dyana Rose created FLINK-8439:
---------------------------------

             Summary: Document using a custom AWS Credentials Provider with flink-3s-fs-hadoop
                 Key: FLINK-8439
                 URL: https://issues.apache.org/jira/browse/FLINK-8439
             Project: Flink
          Issue Type: Improvement
          Components: Documentation
            Reporter: Dyana Rose


This came up when using the s3 for the file system backend and running under ECS.

With no credentials in the container, hadoop-aws will default to EC2 instance level credentials when accessing S3. However when running under ECS, you will generally want to default to the task definition's IAM role.

In this case you need to set the hadoop property
{code:java}
fs.s3a.aws.credentials.provider{code}
to a fully qualified class name(s). see [hadoop-aws docs|https://github.com/apache/hadoop/blob/1ba491ff907fc5d2618add980734a3534e2be098/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md]

This works as expected when you add this setting to flink-conf.yaml but there is a further 'gotcha.'  Because the AWS sdk is shaded, the actual full class name for, in this case, the ContainerCredentialsProvider is
{code:java}
org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider{code}
 

meaning the full setting is:
{code:java}
fs.s3a.aws.credentials.provider: org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider{code}
If you instead set it to the unshaded class name you will see a very confusing error stating that the ContainerCredentialsProvider doesn't implement AWSCredentialsProvider (which it most certainly does.)

Adding this information (how to specify alternate Credential Providers, and the name space gotcha) to the [AWS deployment docs|https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/aws.html] would be useful to anyone else using S3.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)