You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/07/27 10:57:15 UTC
[GitHub] [iceberg] singhpk234 opened a new pull request, #5366: AWS: Call abortUpload only once when any of the completable future fails
singhpk234 opened a new pull request, #5366:
URL: https://github.com/apache/iceberg/pull/5366
### About the Change :
Presently whenever a failure is observed in any of the completableFutures in `uploadParts()`, it causes further completable futures fail with stagings files deleted as part of `abortUpload` which result in FileNotFoundException when it tries to create a request for it using `RequestBody.fromFile(file)`
https://github.com/apache/iceberg/blob/7a92bf5e31a995c61dc598d2a36f598d5c120756/aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java#L306
I think this is what was also highlighted in @stevenzwu 's observation here (https://github.com/apache/iceberg/issues/4168#issuecomment-1123025380)
This change attempts :
* To avoid unnecessary deletes being called for all staging files on every subsequent `completableFutures` which fail due to fileNotFoundException.
* Cancels the remaining futures in which would not be required as the abort is called, reduces number of calls to abort multi-part upload as well.
### Testing Done
Existing UT's + a UT to see abort is called only once rather than atleast once.
cc @rdblue @jackye1995 @RussellSpitzer @stevenzwu @amogh-jahagirdar @rajarshisarkar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] singhpk234 commented on pull request #5366: AWS: Call abortUpload only once when any of the completable future fails
Posted by GitBox <gi...@apache.org>.
singhpk234 commented on PR #5366:
URL: https://github.com/apache/iceberg/pull/5366#issuecomment-1202056847
Thanks @RussellSpitzer !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] singhpk234 commented on pull request #5366: AWS: Call abortUpload only once when any of the completable future fails
Posted by GitBox <gi...@apache.org>.
singhpk234 commented on PR #5366:
URL: https://github.com/apache/iceberg/pull/5366#issuecomment-1198887560
This PR can also fix the flakyness of TestS3OutputStream#testAbortAfterFailedPartUpload in recent runs of master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #5366: AWS: Call abortUpload only once when any of the completable future fails
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on PR #5366:
URL: https://github.com/apache/iceberg/pull/5366#issuecomment-1201566177
Thanks @singhpk234 and thanks @saswata-dutta for review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer merged pull request #5366: AWS: Call abortUpload only once when any of the completable future fails
Posted by GitBox <gi...@apache.org>.
RussellSpitzer merged PR #5366:
URL: https://github.com/apache/iceberg/pull/5366
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org