[jira] [Created] (FLINK-32375) Flink AWS Source AssumeRole in VPC

["Tomas Witzany (Jira)" <ji...@apache.org>]

Tomas Witzany created FLINK-32375:
-------------------------------------

             Summary: Flink AWS Source AssumeRole in VPC
                 Key: FLINK-32375
                 URL: https://issues.apache.org/jira/browse/FLINK-32375
             Project: Flink
          Issue Type: New Feature
          Components: Connectors / AWS
    Affects Versions: 1.15.4
         Environment: Description of setup:
 * Flink 1.15.4
 * running on Amazon KDA (managed flink)
 * runtime is running inside a VPC
 * input stream cross-account
            Reporter: Tomas Witzany


Current way to configure auth against AWS supports assuming a role, but when you assume a role in a VPC without a NAT gateway, the global STS endpoint is not accessible. And there is no way to configure the provider to use a different endpoint.

This means that there currently is no supported way to configure AWS auth in such a situation. Note that you can add an sts endpoint to a VPC, but its always a regional endpoint, not the global endpoint.

Options on how you can configure this:
 * configuring the aws DefaultsMode, by default legacy, to in-region:
 ** environment variables - not possible in KDA
 ** system variables - not possible in KDA
 * adding endpoint configuration options to the assume role provider

The piece of code that creates the provider and how it could be extended to support endpoint configuration

 

```

return StsAssumeRoleCredentialsProvider.builder()
.refreshRequest(
AssumeRoleRequest.builder()
.roleArn(
configProps.getProperty(
AWSConfigConstants.roleArn(configPrefix)))
.roleSessionName(
configProps.getProperty(
AWSConfigConstants.roleSessionName(configPrefix)))
.externalId(
configProps.getProperty(
AWSConfigConstants.externalId(configPrefix)))
.build())
.stsClient(
StsClient.builder()
.credentialsProvider(
getCredentialsProvider(
configProps,
AWSConfigConstants.roleCredentialsProvider(
configPrefix)))
.endpointOverride(new URI( //added code
configProps.getProperty(AWSConfigConstants.endpointOverride(configPrefix)) // added code
)) // added code
.region(getRegion(configProps))
.build())
.build();

```

 

I am not entirely certain that there is no other way to configure this in my situation, my current plan is to build my own version of the connectors with this option supported. If a feature like this would be nice to have, I would be happy to share my results in a PR afterwards.

However,  if there is a better way to configure this, I would be happy to hear about it. If you know of some trick to do this in KDA, where you have limited options to configure things.

Some other pats to attack this problem:
 * trying to set system properties on the task manager before the kinesis source is initialized - this is hard as you dont have control over execution order, probably doable though with some hacks
 * ask AWS support to set a system property with flink config file options - this is hard as it will involve aws support



--
This message was sent by Atlassian Jira
(v8.20.10#820010)