[jira] [Commented] (HADOOP-13674) S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.

["Chris Nauroth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543380#comment-15543380 ] 

Chris Nauroth commented on HADOOP-13674:
----------------------------------------

Here is the error I see when I attempt to access a bucket in us-west-2, but with {{fs.s3a.endpoint}} pointing to us-west-1.

{code}
> hadoop fs -D fs.s3a.endpoint=s3-us-west-1.amazonaws.com -ls s3a://cnauroth-test-aws-s3a-logs/
ls: getFileStatus on : com.amazonaws.services.s3.model.AmazonS3Exception: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: EC6C7FCF8B40B27C), S3 Extended Request ID: EQ1h4MW2CRLV4ZJBGs2xz2CVXwsfGS5X+ByWfyl1tdzbXbf7bFn5DI5pejcWWCmu1/P/uDEOjaU=: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: EC6C7FCF8B40B27C)
{code}

It says that the endpoint is wrong, but it doesn't say which endpoint is correct.  Turning on debug logging shows that the information does come back in the HTTP 301 response:

{code}
> hadoop --loglevel DEBUG fs -D fs.s3a.endpoint=s3-us-west-1.amazonaws.com -ls s3a://cnauroth-test-aws-s3a-logs/
...
16/10/03 13:35:28 DEBUG http.wire:  << "<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>cnauroth-test-aws-s3a-logs</Bucket><Endpoint>cnauroth-test-aws-s3a-logs.s3-us-west-2.amazonaws.com</Endpoint><RequestId>995927D9C5DD8F90</RequestId><HostId>LK/kvbR/gdnxyr5JXj1L41TOfcO4VBF6MtT8FkwOXXyRdjhasccrHc2bux+b4uHSqJmiBEgHJcI=</HostId></Error>"
...
{code}

It appears that the AWS SDK maps the {{<Endpoint>}} element into the map returned by [{{AmazonS3Exception#getAdditionalDetails()}}|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/AmazonS3Exception.html#getAdditionalDetails--].  We can use that to get the information and put it into the exception thrown from S3A.

> S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-13674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13674
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>            Priority: Minor
>
> When accessing the S3 service through a region-specific endpoint, the bucket must be located in that region.  If the client attempts to access a bucket that is not located in that region, then the service replies with a 301 redirect and the correct region endpoint.  However, the exception thrown by S3A does not include the correct endpoint.  If we included that information in the exception, it would make it easier for users to diagnose and fix incorrect configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org