You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Feifan Wang (Jira)" <ji...@apache.org> on 2022/06/21 10:17:00 UTC

[jira] [Created] (FLINK-28172) Scatter dstl files into separate directories by job id

Feifan Wang created FLINK-28172:
-----------------------------------

             Summary: Scatter dstl files into separate directories by job id
                 Key: FLINK-28172
                 URL: https://issues.apache.org/jira/browse/FLINK-28172
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / State Backends
    Affects Versions: 1.15.0
            Reporter: Feifan Wang


In the current implementation of {_}FsStateChangelogStorage{_}, dstl files from all jobs are put into the same directory (configured via {_}dstl.dfs.base-path{_}). Everything is fine if it's a filesystem like S3.But if it is a file system like hadoop, there will be some problems.

First, there may be an upper limit to the number of files in a single directory. Increasing this threshold will greatly reduce the performance of the distributed file system.

Second, dstl file management becomes difficult because the user cannot tell which job the dstl file belongs to, especially when the retained checkpoint is turned on.
h3. Propose
 # create a subdirectory named with the job id under the _dstl.dfs.base-path_ directory when the job starts
 # all dstl files upload to the subdirectory

( Going a step further, we can even create two levels of subdirectories under the _dstl.dfs.base-path_ directory, like _base-path/\{jobId}/dstl ._ This way, if the user configures the same dstl.dfs.base-path as state.checkpoints.dir, all files needed for job recovery will be in the same directory and well organized. )



--
This message was sent by Atlassian Jira
(v8.20.7#820007)