You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Tao Li (Jira)" <ji...@apache.org> on 2021/01/24 04:35:00 UTC
[jira] [Issue Comment Deleted] (BEAM-10335) Add STS Assume role
credentials provider to AwsModule
[ https://issues.apache.org/jira/browse/BEAM-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tao Li updated BEAM-10335:
--------------------------
Comment: was deleted
(was: Hi [~jalmeida] thanks for contributing to this nice feature to enable assume role. I am doing some testing against this feature by specifying an IAM role and access s3 data. I am using:
# beam 2.25 version
# direct runner for the testing.
# ParquetIO to read parquet files
Below is the command.
java -cp <jar file> --inputPath="s3://output/path/*.parquet" --awsCredentialsProvider='\{"@type":"STSAssumeRoleSessionCredentialsProvider", "roleArn":"<iam role name>", "roleSessionName":"<session name>"}'
Is this the right way to specify the role? I am seeing a potential problem with this command. I have made sure the specified IAM role has read access to the s3 files (I can access these files by using aws sdk directly). But I am seeing below error with the java command above. Please advise. Thanks!
Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 8E7E2BFE1F794A36; S3 Extended Request ID: w5Vqpj3rzt3OxRQXyhXNxpJLI/AoVn2Q9v0vvQFHFvKAh3yBvtOGEFiC9m3CeZlDZ3fVhKv/qBQ=), S3 Extended Request ID: w5Vqpj3rzt3OxRQXyhXNxpJLI/AoVn2Q9v0vvQFHFvKAh3yBvtOGEFiC9m3CeZlDZ3fVhKv/qBQ=Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 8E7E2BFE1F794A36; S3 Extended Request ID: w5Vqpj3rzt3OxRQXyhXNxpJLI/AoVn2Q9v0vvQFHFvKAh3yBvtOGEFiC9m3CeZlDZ3fVhKv/qBQ=), S3 Extended Request ID: w5Vqpj3rzt3OxRQXyhXNxpJLI/AoVn2Q9v0vvQFHFvKAh3yBvtOGEFiC9m3CeZlDZ3fVhKv/qBQ= at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:353) at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:321) at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:216) at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303) at
)
> Add STS Assume role credentials provider to AwsModule
> -----------------------------------------------------
>
> Key: BEAM-10335
> URL: https://issues.apache.org/jira/browse/BEAM-10335
> Project: Beam
> Issue Type: Improvement
> Components: io-java-aws
> Reporter: Julius Almeida
> Assignee: Julius Almeida
> Priority: P2
> Labels: starter
> Fix For: 2.24.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In order to perform multi account s3 write, we need to assume role.
> Current implementation of AwsModule has no options to serialize & deserialize credentials provided by STSAssumeRoleSessionCredentialsProvider.
> Need to add support for STSAssumeRoleSessionCredentialsProvider in AwsModule.
> AwsModule.class : [https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)