[jira] [Created] (FLINK-28699) Native rocksdb full snapshot in non-incremental checkpointing

["Lihe Ma (Jira)" <ji...@apache.org>]

Lihe Ma created FLINK-28699:
-------------------------------

             Summary: Native rocksdb full snapshot in non-incremental checkpointing
                 Key: FLINK-28699
                 URL: https://issues.apache.org/jira/browse/FLINK-28699
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / State Backends
    Affects Versions: 1.15.1, 1.14.5
            Reporter: Lihe Ma


When rocksdb statebackend is used and state.backend.incremental enabled, flink will figure out newly created sst files generated by rocksdb during checkpoint, and read all the states from rocksdb and write to files during savepoint [1].

When state.backend.incremental disabled, flink will read all the states from rocksdb and generate state files in checkpoint and savepoint [2]. This makes sense in savepoint, cause user can take a savepoint with rocksdb statebackend and then restore it using another statebackend, but in checkpoint, deserialisation and serialisation of state results in performance loss.

If the native rocksdb snapshot is introduced in full snapshot, theoretically better performance can be achieved. At the same time, savepoint remains the same as before.

 
 # https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java
 # https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksFullSnapshotStrategy.java



--
This message was sent by Atlassian Jira
(v8.20.10#820010)