You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by sreenath kodedala <ve...@me.com> on 2018/01/11 23:25:43 UTC

Kinesis Connectors - With Temporary Credentials

> 
> Hi,
> 
> According to my understanding, Kinesis Connector requires Access Key and Secret Key to connect.
> 
> Is it possible or any work around to use Temporary Credentials from AWS to use in Kinesis Connector?
> We have scenario where we are trying to access cross-account Stream and we are assuming the role. So, in this scenario we get temporary credentials with a token which will expire every hour. 
> 
> Thank you
> -Sree

Re: Kinesis Connectors - With Temporary Credentials

Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Ah, I see. Temporary Credentials are delegated through the AWS Security Token Service through the AssumeRole API.
Sorry, I wasn’t knowledgable of the Temporary Credentials feature before.

Seems like we should add support for the STSAssumeRoleSessionCredentialsProvider [1]. And yes, your observation is correct that I think this would be a matter of extending the AWSUtil class.

I’ve filed a JIRA for the issue: FLINK-8417 [2]. Would you like to contribute this feature? That would be of great help and I think it’ll be a useful addition. If yes, feel free to ping me for any questions you may have.

Cheers,
Gordon

[1] https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.html
[2] https://issues.apache.org/jira/browse/FLINK-8417

On 12 January 2018 at 7:46:10 AM, sreenath kodedala (vedarad@me.com) wrote:

No, they are not but we can definitely look into that.

If no, is there a workaround to implement or customize AWS Utils?

Thank you

On Jan 11, 2018, at 6:41 PM, Tzu-Li (Gordon) Tai <tz...@apache.org> wrote:

Hi Sree,

Are Temporary Credentials automatically shipped with AWS EC2 instances when delegated to the role?
If yes, you should be able to just configure the properties so that the Kinesis consumer automatically fetches credentials from the AWS instance.
To do that, simply do not provide the Access Key and Secret Key explicitly in the properties, and it will use the above default behaviour.

Apparently, the Kinesis connector docs [1] do not educate this preferred default behavior well enough. I’ll file a JIRA to improve that.

Cheers,
Gordon

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/connectors/kinesis.html
On 12 January 2018 at 7:25:58 AM, sreenath kodedala (vedarad@me.com) wrote:


>  
> Hi, 
>  
> According to my understanding, Kinesis Connector requires Access Key and Secret Key to connect. 
>  
> Is it possible or any work around to use Temporary Credentials from AWS to use in Kinesis Connector? 
> We have scenario where we are trying to access cross-account Stream and we are assuming the role. So, in this scenario we get temporary credentials with a token which will expire every hour.  
>  
> Thank you 
> -Sree

Re: Kinesis Connectors - With Temporary Credentials

Posted by sreenath kodedala <ve...@me.com>.
No, they are not but we can definitely look into that.

If no, is there a workaround to implement or customize AWS Utils?

Thank you

> On Jan 11, 2018, at 6:41 PM, Tzu-Li (Gordon) Tai <tz...@apache.org> wrote:
> 
> Hi Sree,
> 
> Are Temporary Credentials automatically shipped with AWS EC2 instances when delegated to the role?
> If yes, you should be able to just configure the properties so that the Kinesis consumer automatically fetches credentials from the AWS instance.
> To do that, simply do not provide the Access Key and Secret Key explicitly in the properties, and it will use the above default behaviour.
> 
> Apparently, the Kinesis connector docs [1] do not educate this preferred default behavior well enough. I’ll file a JIRA to improve that.
> 
> Cheers,
> Gordon
> 
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/connectors/kinesis.html <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/connectors/kinesis.html>
> On 12 January 2018 at 7:25:58 AM, sreenath kodedala (vedarad@me.com <ma...@me.com>) wrote:
> 
>> 
>> >  
>> > Hi, 
>> >  
>> > According to my understanding, Kinesis Connector requires Access Key and Secret Key to connect. 
>> >  
>> > Is it possible or any work around to use Temporary Credentials from AWS to use in Kinesis Connector? 
>> > We have scenario where we are trying to access cross-account Stream and we are assuming the role. So, in this scenario we get temporary credentials with a token which will expire every hour.  
>> >  
>> > Thank you 
>> > -Sree

Re: Kinesis Connectors - With Temporary Credentials

Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Sree,

Are Temporary Credentials automatically shipped with AWS EC2 instances when delegated to the role?
If yes, you should be able to just configure the properties so that the Kinesis consumer automatically fetches credentials from the AWS instance.
To do that, simply do not provide the Access Key and Secret Key explicitly in the properties, and it will use the above default behaviour.

Apparently, the Kinesis connector docs [1] do not educate this preferred default behavior well enough. I’ll file a JIRA to improve that.

Cheers,
Gordon

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/connectors/kinesis.html
On 12 January 2018 at 7:25:58 AM, sreenath kodedala (vedarad@me.com) wrote:


>  
> Hi,  
>  
> According to my understanding, Kinesis Connector requires Access Key and Secret Key to connect.  
>  
> Is it possible or any work around to use Temporary Credentials from AWS to use in Kinesis Connector?  
> We have scenario where we are trying to access cross-account Stream and we are assuming the role. So, in this scenario we get temporary credentials with a token which will expire every hour.  
>  
> Thank you  
> -Sree