S3 Input/Output with temporary credentials (IAM Roles)

[Vladimir Stoyak <vs...@yahoo.com>]

Our setup involves AWS IAM roles when with permanent access_key and access_secret we need to assume specific role (ie getting temporary credentials to use AWS resources).
I was wondering what would be the best way handling this, ie how to set fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey programmatically and also how to handle expired sessions.
Thanks,Vladimir

Re: S3 Input/Output with temporary credentials (IAM Roles)

[Robert Metzger <rm...@apache.org>]

Hi Vladimir,

Flink is using Hadoop's S3 File System implementation. It seems that this
feature is not supported by their implementation:
https://issues.apache.org/jira/browse/HADOOP-9680
This issue contains some more information:
https://issues.apache.org/jira/browse/HADOOP-9384 It seems that the s3a
implementation is the one where to implement the feature.

And realistically, I don't see Hadoop fixing this in the foreseeable future
:)

I see the following options:
- We could try adding the feature to Hadoop's s3a implementation. It'll
probably be a few months until the fix is reviewed, merged and released
(but you could probably extract the relevant code into your own project and
run it from there)
- You implement an s3 file system implementation for Flink with the
required features (that is not as hard as it sounds).

Sorry that I can not give you a better solution for this.

Regards,
Robert


On Fri, Dec 11, 2015 at 3:42 PM, Vladimir Stoyak <vs...@yahoo.com> wrote:

> Our setup involves AWS IAM roles when with permanent access_key and
> access_secret we need to assume specific role (ie getting temporary
> credentials to use AWS resources).
>
> I was wondering what would be the best way handling this, ie how to set fs.s3n.awsAccessKeyId
> and fs.s3n.awsSecretAccessKey programmatically and also how to handle
> expired sessions.
>
> Thanks,
> Vladimir
>