[jira] [Created] (FLINK-40595) Native S3 always aborts multipart uploads required for checkpoint recovery

["Mate Czagany (Jira)" <ji...@apache.org>]

Mate Czagany created FLINK-40595:
------------------------------------

             Summary: Native S3 always aborts multipart uploads required for checkpoint recovery
                 Key: FLINK-40595
                 URL: https://issues.apache.org/jira/browse/FLINK-40595
             Project: Flink
          Issue Type: Bug
          Components: FileSystems
            Reporter: Mate Czagany


`NativeS3RecoverableFsDataOutputStream.close()` unconditionally aborts its multipart upload, including after `persist()` has returned recoverable state referencing that upload.

 

This method is called during task shutdown or cancellation to dispose an in-progress part file. Restoring from a checkpoint will re-use the aborted upload ID, leading to a failure of `NoSuchUpload`. 

 

My suggestion is to not abort the MPUs, and accept that there will be leaked MPUs, just like it is with the Hadoop connector right now. These uploads will require external cleanup, such as an S3 lifecycle policy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)