[jira] [Commented] (HADOOP-18095) s3a connector to fully support AWS partitions,

["Steve Loughran (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513502#comment-17513502 ] 

Steve Loughran commented on HADOOP-18095:
-----------------------------------------

* whatever is needed for the sse tests to be happy...wildcards are fine.
* adding a partition field to the StoreContext too. is there any way to dynamically determine this, or will it just need to be configured

DelegationTokens are trouble, but given the lack of complaints I'm not sure how many people use that assumed role feature in production. (we use DTs and the plugin point, but with our own tokens and auth service). 

ideally all test configs should be retargetable, as people doing private store testing need this. generally the more aws-specific features (STS, SSE) are skipped. If there are some paths/buckets/endpoints which aren't,. thats something to fix

i like the idea of a pubilc dataset class incorporating all of this. we could have something which takes a path to a hadoop XML config file; we'd have a default resource but the maven build could be pointed at another via a command line property. this file could contain all the settings for a test against a partition or internal s3-compatible store

> s3a connector to fully support AWS partitions,
> ----------------------------------------------
>
>                 Key: HADOOP-18095
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18095
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.2
>            Reporter: Steve Loughran
>            Priority: Minor
>
> There are some minor issues in using the S3A connector's more advanced features in china
> see https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html
> Specifically, that "arn:aws:" prefix we use for all arns needs to be configurable so that aws-cn can be used instead.
> This means finding where we create and use these in production code (dynamically creating IAM role policies) and in tests, and making it configurable.  
> proposed
> * add an option {{fs.s3a.aws.partition}}, default aws.
> * new StoreContext methods to query this, and create the arn for the current bucket (string concat or from the bucket's ARN if created with an AP ARN)
> * docs
> I remember ABFS had a problem with oauth endpoints, that was a lot more serious.
> Can't think of real tests for this, other than verifying that if you create an invalid partition "aws-mars" some things break.
> someone needs to run all our existing tests in china, including those with IAM roles and SSE-KMS.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org