[jira] [Assigned] (FLINK-40595) Native S3 always aborts multipart uploads required for checkpoint recovery

["Gabor Somogyi (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-40595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gabor Somogyi reassigned FLINK-40595:
-------------------------------------

    Assignee: Mate Czagany

> Native S3 always aborts multipart uploads required for checkpoint recovery
> --------------------------------------------------------------------------
>
>                 Key: FLINK-40595
>                 URL: https://issues.apache.org/jira/browse/FLINK-40595
>             Project: Flink
>          Issue Type: Bug
>          Components: FileSystems
>            Reporter: Mate Czagany
>            Assignee: Mate Czagany
>            Priority: Major
>              Labels: pull-request-available
>
> `NativeS3RecoverableFsDataOutputStream.close()` unconditionally aborts its multipart upload, including after `persist()` has returned recoverable state referencing that upload.
> This method is called during task shutdown or cancellation to dispose an in-progress part file. Restoring from a checkpoint will re-use the aborted upload ID, leading to a failure of `NoSuchUpload`. 
> My suggestion is to not abort the MPUs, and accept that there will be leaked MPUs, just like it is with the Hadoop connector right now. These uploads will require external cleanup, such as an S3 lifecycle policy.
> The promise of FLIP-555 to "ensure that orphaned multipart uploads are cleaned up" is not really true now anyways, and I don't see any reliable way to make it happen. For now I think we should fix this correctness gap, and then look into how to properly clean up the orphaned MPUs in a separate ticket



--
This message was sent by Atlassian Jira
(v8.20.10#820010)