You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/06/28 07:43:00 UTC
[jira] [Commented] (FLINK-9686) Flink Kinesis Producer: Enable
Kinesis authentication via AssumeRole
[ https://issues.apache.org/jira/browse/FLINK-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526035#comment-16526035 ]
ASF GitHub Bot commented on FLINK-9686:
---------------------------------------
GitHub user fmthoma opened a pull request:
https://github.com/apache/flink/pull/6221
[FLINK-9686] [kinesis] Enable Kinesis authentication via AssumeRole
## What is the purpose of the change
Enable `FlinkKinesisProducer` to authenticate via assuming a role.
### Current situation:
FlinkKinesisProducer can authenticate with Kinesis by retrieving credentials via one of the following mechanisms:
* Environment variables
* System properties
* An AWS profile
* Directly provided credentials (`BASIC`)
* AWS's own default heuristic (`AUTO`)
For streaming across AWS accounts, it is considered good practise to enable access to the remote Kinesis stream via a role, rather than passing credentials for the remote account.
### Proposed change:
Add a new credentials provider specifying a role ARN, session name, and an additional credentials provider supplying the credentials for assuming the role.
Config example for assuming role `<role-arn>` with auto-detected credentials:{{}}
```
aws.credentials.provider: ASSUME_ROLE
aws.credentials.provider.role.arn: <role-arn>
aws.credentials.provider.role.sessionName: my-session-name
aws.credentials.provider.role.provider: AUTO
```
`ASSUME_ROLE` credentials providers can be nested, i.e. it is possible to assume a role which in turn is allowed to assume another role:
```
aws.credentials.provider: ASSUME_ROLE
aws.credentials.provider.role.arn: <role-arn>
aws.credentials.provider.role.sessionName: my-session-name
aws.credentials.provider.role.provider: ASSUME_ROLE
aws.credentials.provider.role.provider.role.arn: <nested-role-arn>
aws.credentials.provider.role.provider.role.sessionName: my-nested-session-name
aws.credentials.provider.role.provider.role.provider: AUTO
```
## Brief change log
- Add `aws.credentials.provider` option `ASSUME_ROLE` for authenticating via assuming a role.
## Verifying this change
The feature changed was not covered by tests, and it is hard to add non-trivial tests. It can be verified manually:
* Create an AWS IAM user and a role, and give the user permissions to assume the role.
* Create a Kinesis stream, and give the role permissions to write to this stream, but not the user.
* Set up the config by passing the user's credentials to assume the role
* The Kinesis producer should now be able to write to the stream.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): yes
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no, although a config option is added.
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: don't know
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs, JavaDocs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fmthoma/flink enableRoles
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6221.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6221
----
commit 45adaa5a14a3db340fce878db50ce6ed8716fe6c
Author: Franz Thoma <fr...@...>
Date: 2018-05-14T13:02:12Z
[FLINK-9686] [kinesis] Allow creating AWS credentials by assuming a role
Config example:
```
aws.credentials.provider: ASSUME_ROLE
aws.credentials.provider.role.arn: <arn>
aws.credentials.provider.role.sessionName: session-name
aws.credentials.provider.role.provider: AUTO
```
commit 3189b48a6cac2898f63cbf75105f544cde32ff58
Author: Franz Thoma <fr...@...>
Date: 2018-05-14T13:04:08Z
[FLINK-9686] [kinesis] Housekeeping: Use early return instead of variable assignment and break
commit 6fe344c8adb9d2c0bed0006216105b0a5032da55
Author: Franz Thoma <fr...@...>
Date: 2018-05-14T13:13:06Z
[FLINK-9686] [kinesis] Housekeeping
commit eb404061e9c4f87dbba1d18a3300fc190af5ea92
Author: Franz Thoma <fr...@...>
Date: 2018-05-15T13:33:50Z
[FLINK-9686] [kinesis] Add dependency on aws-java-sdk-sts
Implicitly (via `Class.forName`) used by `STSProfileCredentialsServiceProvider`.
Due to shading, it is not possible to treat this as a "provided" dependency, as
Maven rewrites the class name with the shaded one, which would force clients to
provide aws-java-sdk-sts shaded in the same way.
commit d7ef8b977f379b7178260cbaf7bcdde6b6b3df1a
Author: Franz Thoma <fr...@...>
Date: 2018-06-28T07:42:20Z
[FLINK-9686] [kinesis] Mention new config option in docs
----
> Flink Kinesis Producer: Enable Kinesis authentication via AssumeRole
> --------------------------------------------------------------------
>
> Key: FLINK-9686
> URL: https://issues.apache.org/jira/browse/FLINK-9686
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
> Reporter: Franz Thoma
> Priority: Major
> Labels: pull-request-available
>
> h2. Current situation:
> FlinkKinesisProducer can authenticate with Kinesis by retrieving credentials via one of the following mechanisms:
> * Environment variables
> * System properties
> * An AWS profile
> * Directly provided credentials (\{{BASIC}})
> * AWS's own default heuristic (\{{AUTO}})
> For streaming across AWS accounts, it is considered good practise to enable access to the remote Kinesis stream via a role, rather than passing credentials for the remote account.
> h2. Proposed change:
> Add a new credentials provider specifying a role ARN, session name, and an additional credentials provider supplying the credentials for assuming the role.
> Config example for assuming role {{<role-arn>}} with auto-detected credentials:{{}}
> {code:java}
> aws.credentials.provider: ASSUME_ROLE
> aws.credentials.provider.role.arn: <role-arn>
> aws.credentials.provider.role.sessionName: my-session-name
> aws.credentials.provider.role.provider: AUTO
> {code}
> {{ASSUME_ROLE}} credentials providers can be nested, i.e. it is possible to assume a role which in turn is allowed to assume another role:
> {code:java}
> aws.credentials.provider: ASSUME_ROLE
> aws.credentials.provider.role.arn: <role-arn>
> aws.credentials.provider.role.sessionName: my-session-name
> aws.credentials.provider.role.provider: ASSUME_ROLE
> aws.credentials.provider.role.provider.role.arn: <nested-role-arn>
> aws.credentials.provider.role.provider.role.sessionName: my-nested-session-name
> aws.credentials.provider.role.provider.role.provider: AUTO
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)