Replacing AWS credentials provider for S3 filesystem

[Rafi Aroch <ra...@gmail.com>]

Hi,

I'm seeing an issue when trying to set a different credentials provider for
AWS S3.

I'm setting in flink-conf.json the *fs.s3a.aws.credentials.provider *to a
different value.
I'm using the *flink-s3-fs-hadoop* dependency and I get an exception when
job starts:

*RuntimeException: Failed to start checkpoint ID counter: From option
fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class
org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider
not found*

I took a look at the code at *org.apache.flink.fs.s3hadoop.S3FileSystemFactory
[1]* and saw that the shaded prefix is set to:
*org.apache.flink.fs.s3hadoop.shaded*

But in the pom.xml file I see that the AWS packages are migrated to:
*org.apache.flink.fs.s3base.shaded.com.amazon
[2]*

I saw there was work done in this area here: FLINK-8439
<https://issues.apache.org/jira/browse/FLINK-8439> commit
<https://github.com/apache/flink/commit/ac97d46ea4586e41ee8606b5c6b6ab6640f6a05a>

Is this a bug in the shaded prefix? Or am I missing something?

[1]
https://github.com/apache/flink/blob/dac36482e2bc7b16d0c1e8dc8ae986d5cf6c1db6/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java#L48
[2]
https://github.com/apache/flink/blob/dac36482e2bc7b16d0c1e8dc8ae986d5cf6c1db6/flink-filesystems/flink-s3-fs-hadoop/pom.xml#L105

Thanks,
Rafi