[jira] [Commented] (FLINK-27132) CheckpointResourcesCleanupRunner might discard shared state of the initial checkpoint

["Matthias Pohl (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-27132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520497#comment-17520497 ] 

Matthias Pohl commented on FLINK-27132:
---------------------------------------

We have some issues with the checkpoint cleanup in general which we're covering right now in FLINK-26606. Right now, your observation is correct that a subsumed Checkpoint is removed from the {{StateHandleStore}} (i.e. what you're referring to as ZK; I'm just mentioning it here because the k8s setup should have the same problem). Any Checkpoint that's not listed in the {{StateHandleStore}} is not subject to removal after the job is terminated globally.

AFAIU, the {{SharedStateRegistry}} only loads the data from {{CompletedCheckpointStore}}, i.e. if the data is not present through these checkpoints, it won't be discarded.

> CheckpointResourcesCleanupRunner might discard shared state of the initial checkpoint
> -------------------------------------------------------------------------------------
>
>                 Key: FLINK-27132
>                 URL: https://issues.apache.org/jira/browse/FLINK-27132
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Checkpointing
>    Affects Versions: 1.15.0, 1.16.0
>            Reporter: Roman Khachatryan
>            Priority: Major
>
> When considering the following case: # A job starts from a checkpoint in NO_CLAIM mode, with incremental checkpoints enabled
>  # It produces some new checkpoints and subsumes the original one (not discarding shared state - before FLINK-24611 or after FLINK-26985)
>  # Job terminates abruptly
>  # The cleaner is started for that job
>  # ZK doesn't have the initial checkpoint, so the store will load only the new checkpoints (created in 2). Shared state is registered
>  # The store is shut down - discarding all the checkpoints and also any shared state
>  
> In 6, if some checkpoint uses the initial state, it will also be discarded
>  
> [~mapohl] could you please confirm this?
>  
> cc: [~yunta]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)