[jira] [Created] (HADOOP-18095) s3a connector to fully support AWS partitions,

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created HADOOP-18095:
---------------------------------------

             Summary: s3a connector to fully support AWS partitions,
                 Key: HADOOP-18095
                 URL: https://issues.apache.org/jira/browse/HADOOP-18095
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 3.3.2
            Reporter: Steve Loughran


There are some minor issues in using the S3A connector's more advanced features in china

see https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html

Specifically, that "arn:aws:" prefix we use for all arns needs to be configurable so that aws-cn can be used instead.

This means finding where we create and use these in production code (dynamically creating IAM role policies) and in tests, and making it configurable.  

proposed
* add an option {{fs.s3a.aws.partition}}, default aws.
* new StoreContext methods to query this, and create the arn for the current bucket (string concat or from the bucket's ARN if created with an AP ARN)
* docs

I remember ABFS had a problem with oauth endpoints, that was a lot more serious.

Can't think of real tests for this, other than verifying that if you create an invalid partition "aws-mars" some things break.

someone needs to run all our existing tests in china, including those with IAM roles and SSE-KMS.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org