[jira] [Created] (FLINK-34982) FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State

["Jinzhong Li (Jira)" <ji...@apache.org>]

Jinzhong Li created FLINK-34982:
-----------------------------------

             Summary: FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State
                 Key: FLINK-34982
                 URL: https://issues.apache.org/jira/browse/FLINK-34982
             Project: Flink
          Issue Type: New Feature
          Components: Runtime / Checkpointing, Runtime / State Backends
            Reporter: Jinzhong Li
             Fix For: 2.0.0


This is a sub-FLIP for the disaggregated state management and its related work, please read the [FLIP-423|https://cwiki.apache.org/confluence/x/R4p3EQ] first to know the whole story.

As outlined in [FLIP-423|https://cwiki.apache.org/confluence/x/R4p3EQ] [1] and [FLIP-427|https://cwiki.apache.org/confluence/x/T4p3EQ] [2], we proposed to disaggregate StateManagement and introduced a disaggregated state storage named ForSt, which evolves from RocksDB. Within the new framework, where the primary storage is placed on the remote file system, several challenges emerge when attempting to reuse the existing fault-tolerance mechanisms of local RocksDB:
 * Because most remote file system don't support hard-link, ForSt can't utilize hard-link to capture a consistent snapshot during checkpoint synchronous phase as rocksdb currently does.
 * The existing file transfer mechanism within RocksDB is inefficient during checkpoints; it involves first downloading the remote working state data to local memory and then uploading it to the checkpoint directory. Likewise, both restore and rescale face the similar problems due to superfluous data transmission.

In order to solve the above problems and improve checkpoint/restore/rescaling performance of disaggregated storage, this FLIP proposes:
 # A new checkpoint strategy for disaggregated state storage: leverage RocksDB's low-level api to retain a consistent snapshot during the checkpoint synchronous phase; and then transfer the snapshot files to checkpoint directory during asynchronous phase;
 # Accelerating checkpoint/restore/rescaling by leverage fast-duplication of remote file system, which can bypass the local TaskManager when transferring data between remote working directory and checkpoint directory.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)