[jira] [Commented] (FLINK-13856) Reduce the delete file api when the checkpoint is completed

["Stephan Ewen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-13856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969289#comment-16969289 ] 

Stephan Ewen commented on FLINK-13856:
--------------------------------------

[~andrew_lin] S3 FS is faking recursive deletes. It makes a call to enumerate all keys with a certain prefix and then deletes them one by one.
It is actually worse than directly deleting the keys, because it needs the additional list command, and it may even be inconsistent, because list is not guaranteed to show all keys in S3.

> Reduce the delete file api when the checkpoint is completed
> -----------------------------------------------------------
>
>                 Key: FLINK-13856
>                 URL: https://issues.apache.org/jira/browse/FLINK-13856
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Checkpointing, Runtime / State Backends
>    Affects Versions: 1.8.1, 1.9.0
>            Reporter: andrew.D.lin
>            Assignee: andrew.D.lin
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: after.png, before.png, f6cc56b7-2c74-4f4b-bb6a-476d28a22096.png
>
>   Original Estimate: 48h
>          Time Spent: 10m
>  Remaining Estimate: 47h 50m
>
> When the new checkpoint is completed, an old checkpoint will be deleted by calling CompletedCheckpoint.discardOnSubsume().
> When deleting old checkpoints, follow these steps:
> 1, drop the metadata
> 2, discard private state objects
> 3, discard location as a whole
> In some cases, is it possible to delete the checkpoint folder recursively by one call?
> As far as I know the full amount of checkpoint, it should be possible to delete the folder directly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)