You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/03/13 11:32:42 UTC
[jira] [Commented] (FLINK-6014) Allow the registration of state
objects in checkpoints
[ https://issues.apache.org/jira/browse/FLINK-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907285#comment-15907285 ]
ASF GitHub Bot commented on FLINK-6014:
---------------------------------------
GitHub user shixiaogang opened a pull request:
https://github.com/apache/flink/pull/3524
[FLINK-6014][checkpoint] Allow the registration of state objects in checkpoints
- Introduce `CompositeStateHandle` which is composed of a collection of `StateObject`s and can register these `StateObject`s in `StateRegistry`.
- Use `StateRegistry` to maintain the reference count of the `StateObject`s in completed checkpoints. A `StateObject` is deleted only if its reference count is zero.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shixiaogang/flink flink-6014
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3524.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3524
----
commit a1796f2a0538c02a2154459fef8f931e14e18fca
Author: xiaogang.sxg <xi...@alibaba-inc.com>
Date: 2017-03-13T11:23:47Z
Allow registration of state objects in checkpoints
commit de272a1f78356b940a3b9421dad70937596c3a94
Author: xiaogang.sxg <xi...@alibaba-inc.com>
Date: 2017-03-13T11:29:21Z
remove unnecessary synchronization in StateRegistry
----
> Allow the registration of state objects in checkpoints
> ------------------------------------------------------
>
> Key: FLINK-6014
> URL: https://issues.apache.org/jira/browse/FLINK-6014
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
> Reporter: Xiaogang Shi
> Assignee: Xiaogang Shi
>
> This issue is the very first step towards incremental checkpointing. We introduce a new state handle named {{CompositeStateHandle}} to be the base of the snapshots taken by task components. Known implementation may include {{KeyedStateHandle}} (for {{KeyedStateBackend}}s), {{SubtaskState}} (for subtasks, splits of {{JobVertex}}) and {{TaskState}} (for {{JobVertex}}s).
> Each {{CompositeStateHandle}} is composed of a collection of {{StateObject}s. It should register all its state objects in {{StateRegistry}} when its checkpoint is added into {{CompletedCheckpointStore}} (i.e., a pending checkpoint completes or a complete checkpoint is reloaded in the recovery).
> When a completed checkpoint is moved out of the {{CompletedCheckpointStore}}, we should not simply discard all state objects in the checkpoint. With the introduction of incremental checkpointing, a {{StateObject}} may be referenced by different checkpoints. We should unregister all the state objects contained in the {{StateRegistry}} first. Only those state objects that are not referenced by any checkpoint can be deleted.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)