You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2019/09/13 13:29:00 UTC
[jira] [Commented] (HADOOP-16573) IAM role created by S3A DT
doesn't include DynamoDB scan
[ https://issues.apache.org/jira/browse/HADOOP-16573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929192#comment-16929192 ]
Steve Loughran commented on HADOOP-16573:
-----------------------------------------
{code}
ard"
2019-09-13 14:22:38,687 [main] INFO s3guard.S3GuardTool (S3GuardTool.java:initMetadataStore(323)) - Metadata store DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new, tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new} is initialized.
2019-09-13 14:22:38,708 [main] INFO s3guard.DynamoDBMetadataStore (DurationInfo.java:<init>(72)) - Starting: Pruning DynamoDB Store
2019-09-13 14:22:38,766 [main] INFO s3guard.DynamoDBMetadataStore (DurationInfo.java:close(87)) - Pruning DynamoDB Store: duration 0:00.058s
java.nio.file.AccessDeniedException: /hwdev-steve-ireland-new: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::980678866538:assumed-role/stevel-s3guard/89db9060-6066-4f84-af7c-a40babaacb2e is not authorized to perform: dynamodb:Scan on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID: 6I1ACO9K5DRGJK70M9BDPF834VVV4KQNSO5AEMVJF66Q9ASUAAJG)
at org.apache.hadoop.fs.s3a.S3AUtils.translateDynamoDBException(S3AUtils.java:437)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerPrune(DynamoDBMetadataStore.java:1602)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.prune(DynamoDBMetadataStore.java:1534)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Prune.run(S3GuardTool.java:1133)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:425)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1700)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1709)
Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::980678866538:assumed-role/stevel-s3guard/89db9060-6066-4f84-af7c-a40babaacb2e is not authorized to perform: dynamodb:Scan on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID: 6I1ACO9K5DRGJK70M9BDPF834VVV4KQNSO5AEMVJF66Q9ASUAAJG)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:4279)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4246)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeScan(AmazonDynamoDBClient.java:3040)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.scan(AmazonDynamoDBClient.java:3006)
at com.amazonaws.services.dynamodbv2.document.internal.ScanCollection.firstPage(ScanCollection.java:53)
at com.amazonaws.services.dynamodbv2.document.internal.PageIterator.next(PageIterator.java:45)
at com.amazonaws.services.dynamodbv2.document.internal.IteratorSupport.nextResource(IteratorSupport.java:87)
at com.amazonaws.services.dynamodbv2.document.internal.IteratorSupport.hasNext(IteratorSupport.java:55)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerPrune(DynamoDBMetadataStore.java:1552)
... 6 more
{code}
> IAM role created by S3A DT doesn't include DynamoDB scan
> --------------------------------------------------------
>
> Key: HADOOP-16573
> URL: https://issues.apache.org/jira/browse/HADOOP-16573
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Priority: Minor
>
> You can't run {{s3guard prune}} with role DTs as we don't create it with permissons to do so.
> I think it may actually be useful to have an option where we don't restrict the role. This doesn't just help with debugging, it would let things like SQS integration pick up the creds from S3A.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org