You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Daniel Carl Jones (Jira)" <ji...@apache.org> on 2022/04/06 13:14:00 UTC
[jira] [Updated] (HADOOP-18095) S3A connector to improve support for all AWS partitions
[ https://issues.apache.org/jira/browse/HADOOP-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Carl Jones updated HADOOP-18095:
---------------------------------------
Summary: S3A connector to improve support for all AWS partitions (was: s3a connector to fully support AWS partitions,)
> S3A connector to improve support for all AWS partitions
> -------------------------------------------------------
>
> Key: HADOOP-18095
> URL: https://issues.apache.org/jira/browse/HADOOP-18095
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.2
> Reporter: Steve Loughran
> Priority: Minor
>
> There are some minor issues in using the S3A connector's more advanced features in china
> see https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html
> Specifically, that "arn:aws:" prefix we use for all arns needs to be configurable so that aws-cn can be used instead.
> This means finding where we create and use these in production code (dynamically creating IAM role policies) and in tests, and making it configurable.
> proposed
> * add an option {{fs.s3a.aws.partition}}, default aws.
> * new StoreContext methods to query this, and create the arn for the current bucket (string concat or from the bucket's ARN if created with an AP ARN)
> * docs
> I remember ABFS had a problem with oauth endpoints, that was a lot more serious.
> Can't think of real tests for this, other than verifying that if you create an invalid partition "aws-mars" some things break.
> someone needs to run all our existing tests in china, including those with IAM roles and SSE-KMS.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org