You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2023/11/21 12:49:00 UTC
[jira] [Commented] (HADOOP-18984) S3A: Support dynamic region resolution
[ https://issues.apache.org/jira/browse/HADOOP-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788407#comment-17788407 ]
Steve Loughran commented on HADOOP-18984:
-----------------------------------------
fyi, cloudstore now has a regions2 command to work this out. The classes we are using are tagged as {{@SdkProtectedApi}} so we shouldn't be using them. but....
maybe we should actually support a chain of region determiners, which go from bucket + endpoint => region
{code}
Determining region using SystemSettingsRegionProvider
=====================================================
System property aws.region
2023-11-21 12:45:44,339 [main] INFO sdk2.Regions2 (StoreDurationInfo.java:<init>(91)) - Starting: SystemSettingsRegionProvider.getRegion()
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2023-11-21 12:45:44,349 [main] INFO sdk2.Regions2 (StoreDurationInfo.java:close(193)) - Duration of SystemSettingsRegionProvider.getRegion(): 0:00:00.011
WARNING: Provider SystemSettingsRegionProvider raised an exception software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or system property (aws.region).
2023-11-21 12:45:44,350 [main] INFO sdk2.Regions2 (Regions2.java:printRegion(101)) - Provider raised an exception
software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or system property (aws.region).
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider.exception(SystemSettingsRegionProvider.java:41)
at java.util.Optional.orElseThrow(Optional.java:290)
at software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider.getRegion(SystemSettingsRegionProvider.java:33)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.printRegion(Regions2.java:98)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.run(Regions2.java:67)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:95)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.exec(Regions2.java:125)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.main(Regions2.java:134)
at regions2.main(regions2.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
region is not known
Determining region using AwsProfileRegionProvider
=================================================
Region info in ~/.aws/config
2023-11-21 12:45:44,353 [main] INFO sdk2.Regions2 (StoreDurationInfo.java:<init>(91)) - Starting: AwsProfileRegionProvider.getRegion()
2023-11-21 12:45:44,380 [main] INFO sdk2.Regions2 (StoreDurationInfo.java:close(193)) - Duration of AwsProfileRegionProvider.getRegion(): 0:00:00.027
Region is determined as "eu-west-2"
Determining region using InstanceMetadataRegionProvider
=======================================================
EC2 metadata; will only work in AWS infrastructure
2023-11-21 12:45:44,380 [main] INFO sdk2.Regions2 (StoreDurationInfo.java:<init>(91)) - Starting: InstanceMetadataRegionProvider.getRegion()
2023-11-21 12:45:45,741 [main] INFO sdk2.Regions2 (StoreDurationInfo.java:close(193)) - Duration of InstanceMetadataRegionProvider.getRegion(): 0:00:01.360
WARNING: Provider InstanceMetadataRegionProvider raised an exception software.amazon.awssdk.core.exception.SdkClientException: Unable to contact EC2 metadata service.
2023-11-21 12:45:45,742 [main] INFO sdk2.Regions2 (Regions2.java:printRegion(101)) - Provider raised an exception
software.amazon.awssdk.core.exception.SdkClientException: Unable to contact EC2 metadata service.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:403)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:414)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:414)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:355)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:351)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getEC2InstanceRegion(EC2MetadataUtils.java:235)
at software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider.tryDetectRegion(InstanceProfileRegionProvider.java:68)
at software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider.getRegion(InstanceProfileRegionProvider.java:52)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.printRegion(Regions2.java:98)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.run(Regions2.java:73)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:95)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.exec(Regions2.java:125)
at org.apache.hadoop.fs.s3a.sdk2.Regions2.main(Regions2.java:134)
at regions2.main(regions2.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:743)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:702)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1600)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1505)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:116)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.doReadResource(EC2MetadataUtils.java:419)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:390)
... 20 more
region is not known
Region found: "eu-west-2"
=========================
Region was determined by AwsProfileRegionProvider as "eu-west-2"
{code}
> S3A: Support dynamic region resolution
> --------------------------------------
>
> Key: HADOOP-18984
> URL: https://issues.apache.org/jira/browse/HADOOP-18984
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> AWS v1 SDK used to do dynamic region inference including EC2 metadata. See HADOOP-17771 for that at work with the chain of: sysprops, env vars, EC2 metadata.
> I think we *try* to do this, but I'm not sure it is working, maybe related to the region=null vs region="" options: its very hard to differentiate the two
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org