[jira] [Commented] (FLINK-13856) Reduce the delete file api when the checkpoint is completed

["Stephan Ewen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-13856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915667#comment-16915667 ] 

Stephan Ewen commented on FLINK-13856:
--------------------------------------

In general, that is a good idea.

However, this is an optimization that we can do only for exclusive state (belongs only to one checkpoint) but not shared state (par of incremental checkpoints).
I remember wanting to do that and there was some issue with selectively deleting the shared state first, and then dropping the exclusive location as a whole.

Furthermore, this only works on proper file systems, not on object stores like S3.

> Reduce the delete file api when the checkpoint is completed
> -----------------------------------------------------------
>
>                 Key: FLINK-13856
>                 URL: https://issues.apache.org/jira/browse/FLINK-13856
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / State Backends
>    Affects Versions: 1.8.1, 1.9.0
>            Reporter: andrew.D.lin
>            Priority: Major
>         Attachments: f6cc56b7-2c74-4f4b-bb6a-476d28a22096.png
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> When the new checkpoint is completed, an old checkpoint will be deleted by calling CompletedCheckpoint.discardOnSubsume().
> When deleting old checkpoints, follow these steps:
> 1, drop the metadata
> 2, discard private state objects
> 3, discard location as a whole
> In some cases, is it possible to delete the checkpoint folder recursively by one call?
> As far as I know the full amount of checkpoint, it should be possible to delete the folder directly.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)