[jira] [Commented] (FLINK-6014) Allow the registration of state objects in checkpoints

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960577#comment-15960577 ] 

ASF GitHub Bot commented on FLINK-6014:
---------------------------------------

Github user shixiaogang commented on the issue:

    https://github.com/apache/flink/pull/3524
  
    @StephanEwen I have updated the PR, making the following changes:
    1. Add a method called `discardSharedStatesOnFail()` in `CompositeStateHandle`.  This method is called when the pending checkpoint fails to complete. That way, we do not need to register shared states once an acknowledge message is received.  All shared states are registered only when the pending checkpoint completes.
    2. Add `SharedStateHandle` and refactor `SharedStateRegistry` as suggested. 
    3. Both registration and unregistration of shared states now are taken place in `CompletedCheckpoint`.
    
    What do you think of these changes?


> Allow the registration of state objects in checkpoints
> ------------------------------------------------------
>
>                 Key: FLINK-6014
>                 URL: https://issues.apache.org/jira/browse/FLINK-6014
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing
>            Reporter: Xiaogang Shi
>            Assignee: Xiaogang Shi
>
> This issue is the very first step towards incremental checkpointing. We introduce a new state handle named {{CompositeStateHandle}} to be the base of the snapshots taken by task components.  Known implementation may include {{KeyedStateHandle}} (for {{KeyedStateBackend}}s), {{SubtaskState}} (for subtasks, splits of {{JobVertex}}) and {{TaskState}} (for {{JobVertex}}s).
> Each {{CompositeStateHandle}} is composed of a collection of {{StateObject}s. It should register all its state objects in {{StateRegistry}} when its checkpoint is added into {{CompletedCheckpointStore}} (i.e., a pending checkpoint completes or a complete checkpoint is reloaded in the recovery). 
> When a completed checkpoint is moved out of the {{CompletedCheckpointStore}}, we should not simply discard all state objects in the checkpoint. With the introduction of incremental checkpointing, a {{StateObject}} may be referenced by different checkpoints. We should unregister all the state objects contained in the {{StateRegistry}} first. Only those state objects that are not referenced by any checkpoint can be deleted.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)