You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/09/29 17:25:00 UTC
[jira] [Commented] (HADOOP-19027) S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
[ https://issues.apache.org/jira/browse/HADOOP-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18023590#comment-18023590 ]
ASF GitHub Bot commented on HADOOP-19027:
-----------------------------------------
steveloughran opened a new pull request, #8003:
URL: https://github.com/apache/hadoop/pull/8003
This is a fraction of #6425 backported to the V1 SDK, which does a lot of V2-specific translation/unwinding.
416 responses are mapped to RangeNotSatisfiableEOFException, whose retry policy is: fail.
Classic EOFException is now retried as connection failure. calls to read() and lazyseek all retry on this with full retry policy, including handling of socket errors.
Everything related to new sdk, its complex http channel stuff: omitted.
This means that the following will be handled
* EOF on socket read
* connection reset on socket read
Out of range GET is mappped to RangeNotSatisfiableEOFException so it is not retried,
this is critical. Without it ITestS3AOpenCost.testReadPastEOF() will time out as it is continually retrying.
### How was this patch tested?
Cherrypick of new ITestS3AOpenCost.
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
> S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
> ----------------------------------------------------------------
>
> Key: HADOOP-19027
> URL: https://issues.apache.org/jira/browse/HADOOP-19027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.5.0, 3.4.1
>
>
> S3AInputStream doesn't seem to recover from Http exceptions raised through HttpClient or through OpenSSL.
> * review the recovery code to make sure it is retrying enough, it looks suspiciously like it doesn't
> * detect the relevant openssl, shaded httpclient and unshaded httpclient exceptions, map to a standard one and treat as comms error in our retry policy
> This is not the same as the load balancer/proxy returning 443/444 which we map to AWSNoResponseException. We can't reuse that as it expects to be created from an {{software.amazon.awssdk.awscore.exception.AwsServiceException}} exception with the relevant fields...changing it could potentially be incompatible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org