[jira] [Commented] (HADOOP-17705) S3A to add option fs.s3a.endpoint.region to set AWS region

["Greg Senia (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-17705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17617276#comment-17617276 ] 

Greg Senia commented on HADOOP-17705:
-------------------------------------

[~stevel@apache.org]is it possible to accept a backport I put together for Hadoop 3.2. I have a few folks who cannot upgrade to Hadoop 3.3.x branch and need the ability to access S3 using V4 signing via VPC endpoints without having to mess with overrides and regex's as we have multiple endpoints in different regions etc.

> S3A to add option fs.s3a.endpoint.region to set AWS region
> ----------------------------------------------------------
>
>                 Key: HADOOP-17705
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17705
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Mehakmeet Singh
>            Assignee: Mehakmeet Singh
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.3.2
>
>          Time Spent: 3h
>  Remaining Estimate: 0h
>
> Currently, AWS region is either constructed via the endpoint URL, by making an assumption that the 2nd component after delimiter "." is the region in endpoint URL, which doesn't work for private links and sets the default to us-east-1 thus causing authorization issue w.r.t the private link.
> The option fs.s3a.endpoint.region allows this to be explicitly set
> h2. how to set the s3 region on older hadoop releases
> For anyone who needs to set the signing region on older versions of the s3a client *you do not need this festure*. instead just provide a custom endpoint to region mapping json file
> # Download the default region mapping file [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
> # Add a new regular expression to map the endpoint/hostname to the target region
> # Save the file as {{/etc/hadoop/conf/awssdk_config_override.json}}
> # verify basic hadop fs -ls commands work
> # copy to the rest of the cluster.
> # There should be no need to restart any services



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org