You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Mukund Thakur (Jira)" <ji...@apache.org> on 2022/08/05 16:05:00 UTC
[jira] [Resolved] (HADOOP-18353) HEAD OBJECT returns only 400 BAD REQUEST when token is expired
[ https://issues.apache.org/jira/browse/HADOOP-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Thakur resolved HADOOP-18353.
------------------------------------
Resolution: Information Provided
> HEAD OBJECT returns only 400 BAD REQUEST when token is expired
> --------------------------------------------------------------
>
> Key: HADOOP-18353
> URL: https://issues.apache.org/jira/browse/HADOOP-18353
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.3
> Reporter: Mukund Thakur
> Priority: Major
>
> I tried reproducing this today by changing this test [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ATemporaryCredentials.java#L116] . Getting a session token for 15 mins and trying every one mins and finally it fails after 15 mins.
>
> Looks like the AWS SDK is not having the Expired Token error message as I could see the same in access logs but I see BadRequest on the SDK logs.
>
> *S3A Connector logs with SDK debug enabled.*
> 2022-07-13 15:44:15,318 [JUnit-testSTS] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(184)) - Using credentials from TemporaryAWSCredentialsProvider
> 2022-07-13 15:44:15,319 [JUnit-testSTS] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1285)) - Sending Request: HEAD [https://mthakur-us-west-1.s3.us-west-1.amazonaws.com|https://mthakur-us-west-1.s3.us-west-1.amazonaws.com/] /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e Headers: (amz-sdk-invocation-id: 41e6e504-1c2b-2701-09bb-ae692dff2515, Content-Type: application/octet-stream, Referer: [https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318], User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10 vendor/Oracle_Corporation cfg/retry-mode/legacy, )
> 2022-07-13 15:44:15,623 [JUnit-testSTS] DEBUG amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1846)) - \{*}Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID{*}: WMGQ0KC4MHEMZTQC; S3 Extended Request ID: IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=; Proxy: null), S3 Extended Request ID: IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
> 2022-07-13 15:44:15,624 [JUnit-testSTS] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:close(3814)) - Filesystem s3a://mthakur-us-west-1 is closed
> *AWS access logs*
> 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 mthakur-us-west-1 [13/Jul/2022:20:44:15 +0000] 67.79.115.98 - WMGQ0KC4MHEMZTQC REST.HEAD.OBJECT test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e "HEAD /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e HTTP/1.1" *400 ExpiredToken* 556 - 5 - "[https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318]" "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10 vendor/Oracle_Corporation cfg/retry-mode/legacy" - IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q= SigV4 ECDHE-RSA-AES128-SHA AuthHeader [mthakur-us-west-1.s3.us-west-1.amazonaws.com|http://mthakur-us-west-1.s3.us-west-1.amazonaws.com/] TLSv1.2 -
>
> I tested by running repeatedly ITestCustomSigner in S3A, and also just ListObjectsV2 on loop… I did just notice your test is failing with HEAD, and *I can reproduce* by running this after credential expiry.
>
> aws s3api head-object --bucket djonesoa-us-west-2 --region us-west-2 --key test-object –debug
>
> To summarise:
> * If I run ListObjectsV2, I get “400 ExpiredToken”{+}{+}{+}{+}
> * If I run HeadObject, I get “400 Bad Request”{+}{+}{+}{+}
> * If I run GetObject, I get “400 ExpiredToken”
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org