You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Prabhu Joseph (Jira)" <ji...@apache.org> on 2022/09/27 11:56:00 UTC
[jira] [Updated] (HADOOP-17705) S3A to add option fs.s3a.endpoint.region to set AWS region
[ https://issues.apache.org/jira/browse/HADOOP-17705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prabhu Joseph updated HADOOP-17705:
-----------------------------------
Description:
Currently, AWS region is either constructed via the endpoint URL, by making an assumption that the 2nd component after delimiter "." is the region in endpoint URL, which doesn't work for private links and sets the default to us-east-1 thus causing authorization issue w.r.t the private link.
The option fs.s3a.endpoint.region allows this to be explicitly set
h2. how to set the s3 region on older hadoop releases
For anyone who needs to set the signing region on older versions of the s3a client *you do not need this festure*. instead just provide a custom endpoint to region mapping json file
# Download the default region mapping file [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
# Add a new regular expression to map the endpoint/hostname to the target region
# Save the file as {{/etc/hadoop/conf/awssdk_config_override.json}}
# verify basic hadop fs -ls commands work
# copy to the rest of the cluster.
# There should be no need to restart any services
was:
Currently, AWS region is either constructed via the endpoint URL, by making an assumption that the 2nd component after delimiter "." is the region in endpoint URL, which doesn't work for private links and sets the default to us-east-1 thus causing authorization issue w.r.t the private link.
The option fs.s3a.endpoint.region allows this to be explicitly set
h2. how to set the s3 region on older hadoop releases
For anyone who needs to set the signing region on older versions of the s3a client *you do not need this festure*. instead just provide a custom endpoint to region mapping json file
# Download the default region mapping file [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
# Add a new regular expression to map the endpoint/hostname to the target region
# Save the file as {{/etc/hadoop/awssdk_config_override.json}}
# verify basic hadop fs -ls commands work
# copy to the rest of the cluster.
# There should be no need to restart any services
> S3A to add option fs.s3a.endpoint.region to set AWS region
> ----------------------------------------------------------
>
> Key: HADOOP-17705
> URL: https://issues.apache.org/jira/browse/HADOOP-17705
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Mehakmeet Singh
> Assignee: Mehakmeet Singh
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, AWS region is either constructed via the endpoint URL, by making an assumption that the 2nd component after delimiter "." is the region in endpoint URL, which doesn't work for private links and sets the default to us-east-1 thus causing authorization issue w.r.t the private link.
> The option fs.s3a.endpoint.region allows this to be explicitly set
> h2. how to set the s3 region on older hadoop releases
> For anyone who needs to set the signing region on older versions of the s3a client *you do not need this festure*. instead just provide a custom endpoint to region mapping json file
> # Download the default region mapping file [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
> # Add a new regular expression to map the endpoint/hostname to the target region
> # Save the file as {{/etc/hadoop/conf/awssdk_config_override.json}}
> # verify basic hadop fs -ls commands work
> # copy to the rest of the cluster.
> # There should be no need to restart any services
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org