You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/03 22:50:15 UTC
[GitHub] [beam] kennknowles opened a new issue, #19142: Add retry logic to S3FileSystem
kennknowles opened a new issue, #19142:
URL: https://github.com/apache/beam/issues/19142
S3FileSystem should have some retry behaviour if ObjectsDelete fails. I have seen such example in our job where 1 item from the delete batch cannot be deleted due to S3 InternalError causing the whole job to restart. The source code I am referring to:
[https://github.com/apache/beam/blob/8a88e72f293ef7f9be6c872aa0dda681458c7ca5/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java#L633](https://github.com/apache/beam/blob/8a88e72f293ef7f9be6c872aa0dda681458c7ca5/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java#L633)
The retry logic might be added to other S3 calls in S3FileSystem as well.
Imported from Jira [BEAM-6031](https://issues.apache.org/jira/browse/BEAM-6031). Original Jira may contain additional context.
Reported by: pawelbartoszek.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche closed issue #19142: Add retry logic to S3FileSystem
Posted by GitBox <gi...@apache.org>.
mosche closed issue #19142: Add retry logic to S3FileSystem
URL: https://github.com/apache/beam/issues/19142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on issue #19142: Add retry logic to S3FileSystem
Posted by GitBox <gi...@apache.org>.
mosche commented on issue #19142:
URL: https://github.com/apache/beam/issues/19142#issuecomment-1250665880
All AWS clients implement automatic retries, this behavior is enabled by default. Adding another level of retries won't do any good...
While possible, it's currently not straight forward to customize the retry policy for the S3 file system. It can be done by setting a custom `s3ClientFactoryClass` in `S3Options`. This factory would then delegate to `ClientBuilderFactory.defaultFactory` with a `ClientConfiguration` containing a customized `RetryConfiguration`.
[https://github.com/apache/beam/blob/5520fe064fc3b7196998d4597746119691eb6681/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/ClientBuilderFactory.java#L151-L155](https://github.com/apache/beam/blob/5520fe064fc3b7196998d4597746119691eb6681/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/ClientBuilderFactory.java#L151-L155)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org