[jira] [Created] (FLINK-24149) Make checkpoint relocatable

["Feifan Wang (Jira)" <ji...@apache.org>]

Feifan Wang created FLINK-24149:
-----------------------------------

             Summary: Make checkpoint relocatable
                 Key: FLINK-24149
                 URL: https://issues.apache.org/jira/browse/FLINK-24149
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Checkpointing
            Reporter: Feifan Wang


h3. Backgroud

FLINK-5763 proposal make savepoint relocatable, checkpoint has similar requirements. For example, to migrate jobs to other HDFS clusters, although it can be achieved through a savepoint, but we prefer to use persistent checkpoints, especially RocksDBStateBackend incremental checkpoints have better performance than savepoint during snapshot and restore.

 

FLINK-8531 standardized directory layout :

 
{code:java}
/user-defined-checkpoint-dir
    |
    + 1b080b6e710aabbef8993ab18c6de98b (job's ID)
        |
        + --shared/
        + --taskowned/
        + --chk-00001/
        + --chk-00002/
        + --chk-00003/
        ...
{code}
 * State backend will create a subdirectory with the job's ID that will contain the actual checkpoints, such as: user-defined-checkpoint-dir/1b080b6e710aabbef8993ab18c6de98b/
 * Each checkpoint individually will store all its files in a subdirectory that includes the checkpoint number, such as: user-defined-checkpoint-dir/1b080b6e710aabbef8993ab18c6de98b/chk-00003/
 * Files shared between checkpoints will be stored in the shared/ directory in the same parent directory as the separate checkpoint directory, such as: user-defined-checkpoint-dir/1b080b6e710aabbef8993ab18c6de98b/shared/
 * Similar to shared files, files owned strictly by tasks will be stored in the taskowned/ directory in the same parent directory as the separate checkpoint directory, such as: user-defined-checkpoint-dir/1b080b6e710aabbef8993ab18c6de98b/taskowned/

h3. Proposal

Since the individually checkpoint directory does not contain complete state data, we cannot make it relocatable, but its parent directory can. The only work left is make the metadata file references relative file paths.

I proposal make these changes to _*FsCheckpointStateOutputStream*_ :
 * introduce _*checkpointDirectory*_ field
 * introduce *_entropyInjecting_* field
 * *_closeAndGetHandle()_* return _*RelativeFileStateHandle*_ with relative path** base on _*checkpointDirectory*_ (except entropy injecting file system)

[~yunta], [~trohrmann] , I verified this in our environment , and I will submit a pull request to accomplish this feature. Please help evaluate whether it is appropriate.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)