[GitHub] [iceberg] jackye1995 commented on pull request #4912: AWS: add retry logic to S3InputStream

[GitBox <gi...@apache.org>]

jackye1995 commented on PR #4912:
URL: https://github.com/apache/iceberg/pull/4912#issuecomment-1146866327

   > Introducing our own retries can ultimately conflict with other retries and can have some bad side-effects (like failures resulting in retries on top of retries).
   
   Yes you are right, I actually plan to add one more PR for configuring retry policy. The issue here is a bit different. I think there are 2 issues that cannot be handled at SDK level:
   1. AWS SDK only handles exceptions up to the AWS client calls, in this case `s3.getObject`. Once you get the input stream, further exceptions are not handled during operations like `read` and `seek`, and that's typically when network issues like connection timeout or socket timeout might happen. I haven't found logic in the SDK that could close a stream, re-seek to the right position and retry a read operation, but maybe I missed some code places.
   2. When closing the input stream, the closing should call `abort` instead of `close` to terminate the HTTP request. This should be done both at retry and stream close time. See [this javadoc](https://github.com/aws/aws-sdk-java-v2/blob/56e21b1d8e18f692ba9193c625661aa786e9946b/core/sdk-core/src/main/java/software/amazon/awssdk/core/ResponseInputStream.java#L29-L31) for more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org