You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Yun Tang (Jira)" <ji...@apache.org> on 2022/08/09 03:27:00 UTC

[jira] [Resolved] (FLINK-28699) Native rocksdb full snapshot in non-incremental checkpointing

     [ https://issues.apache.org/jira/browse/FLINK-28699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yun Tang resolved FLINK-28699.
------------------------------
    Fix Version/s: 1.16.0
       Resolution: Fixed

merged in master: d9a067e5e1c8672930b0ea7d76400a1d3020a1e2

> Native rocksdb full snapshot in non-incremental checkpointing
> -------------------------------------------------------------
>
>                 Key: FLINK-28699
>                 URL: https://issues.apache.org/jira/browse/FLINK-28699
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / State Backends
>    Affects Versions: 1.14.5, 1.15.1
>            Reporter: Lihe Ma
>            Assignee: Lihe Ma
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.16.0
>
>
> When rocksdb statebackend is used and state.backend.incremental enabled, flink will figure out newly created sst files generated by rocksdb during checkpoint, and read all the states from rocksdb and write to files during savepoint [1].
> When state.backend.incremental disabled, flink will read all the states from rocksdb and generate state files in checkpoint and savepoint [2]. This makes sense in savepoint, cause user can take a savepoint with rocksdb statebackend and then restore it using another statebackend, but in checkpoint, deserialisation and serialisation of state results in performance loss.
> If the native rocksdb snapshot is introduced in full snapshot, theoretically better performance can be achieved. At the same time, savepoint remains the same as before.
>  
>  # https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java
>  # https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksFullSnapshotStrategy.java



--
This message was sent by Atlassian Jira
(v8.20.10#820010)