You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/04/29 15:54:04 UTC

[jira] [Commented] (FLINK-6364) Implement incremental checkpointing in RocksDBStateBackend

    [ https://issues.apache.org/jira/browse/FLINK-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989920#comment-15989920 ] 

ASF GitHub Bot commented on FLINK-6364:
---------------------------------------

GitHub user shixiaogang opened a pull request:

    https://github.com/apache/flink/pull/3801

    [FLINK-6364] [checkpoints] Implement incremental checkpointing in RocksDBKeyedStateBackend

    This is the initial implementation of incremental checkpointing in RocksDBKeyedStateBackend. Changes include 
    1. Add a new `CheckpointType` for incremental checkpoints.
    2. Call the `restore()` method for all `KeyedStateBackend` if the restore state handle is null or empty.
    3. Implement `RocksDBIncrementalSnapshotOperation` and `RocksDBIncrementalRestoreOperation` which supports incremental snapshotting and restoring with/without parallelism changes, respectively.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/shixiaogang/flink flink-6364

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3801.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3801
    
----
commit c5340a2186ecfbc48c46048db5adb11af83db511
Author: xiaogang.sxg <xi...@alibaba-inc.com>
Date:   2017-04-29T15:44:36Z

    Implement incremental checkpointing in RocksDBKeyedStateBackend

----


> Implement incremental checkpointing in RocksDBStateBackend
> ----------------------------------------------------------
>
>                 Key: FLINK-6364
>                 URL: https://issues.apache.org/jira/browse/FLINK-6364
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing
>            Reporter: Xiaogang Shi
>            Assignee: Xiaogang Shi
>
> {{RocksDBStateBackend}} is well suited for incremental checkpointing because RocksDB is base on LSM trees,  which record updates in new sst files and all sst files are immutable. By only materializing those new sst files, we can significantly improve the performance of checkpointing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)