[jira] [Created] (FLINK-25524) If enabled changelog, RocksDB incremental checkpoint would always be full

["Yun Tang (Jira)" <ji...@apache.org>]

Yun Tang created FLINK-25524:
--------------------------------

             Summary: If enabled changelog, RocksDB incremental checkpoint would always be full
                 Key: FLINK-25524
                 URL: https://issues.apache.org/jira/browse/FLINK-25524
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Checkpointing, Runtime / State Backends
            Reporter: Yun Tang


Once changelog is enabled, RocksDB incremental checkpoint would only be executed during materialization. During this phase, it will leverage the {{materization id}} as the checkpoint id for RocksDB state backend's snapshot method.

However, current incremental checkpoint mechanism heavily depends on the checkpoint id. And {{SortedMap<Long, Set<StateHandleID>> uploadedStateIDs}} with checkpoint id as the key within {{RocksIncrementalSnapshotStrategy}} is the kernel for incremental checkpoint. Once we notify checkpoint complete of previous checkpoint, it will then remove the uploaded stateIds of that checkpoint, leading to we cannot get proper checkpoint information on the next RocksDBKeyedStateBackend#snapshot. That is to say, we will always upload all RocksDB artifacts.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)