[jira] [Resolved] (FLINK-25524) If enabled changelog, RocksDB incremental checkpoint would always be full

["Roman Khachatryan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Roman Khachatryan resolved FLINK-25524.
---------------------------------------
    Resolution: Fixed

Merged as d0927dd41e2f0441e4e5825ff423dd0e903713f3.

> If enabled changelog, RocksDB incremental checkpoint would always be full
> -------------------------------------------------------------------------
>
>                 Key: FLINK-25524
>                 URL: https://issues.apache.org/jira/browse/FLINK-25524
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Checkpointing, Runtime / State Backends
>    Affects Versions: 1.15.0
>            Reporter: Yun Tang
>            Assignee: Roman Khachatryan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.15.0
>
>
> Once changelog is enabled, RocksDB incremental checkpoint would only be executed during materialization. During this phase, it will leverage the {{materization id}} as the checkpoint id for RocksDB state backend's snapshot method.
> However, current incremental checkpoint mechanism heavily depends on the checkpoint id. And {{SortedMap<Long, Set<StateHandleID>> uploadedStateIDs}} with checkpoint id as the key within {{RocksIncrementalSnapshotStrategy}} is the kernel for incremental checkpoint. Once we notify checkpoint complete of previous checkpoint, it will then remove the uploaded stateIds of that checkpoint, leading to we cannot get proper checkpoint information on the next RocksDBKeyedStateBackend#snapshot. That is to say, we will always upload all RocksDB artifacts.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)