[jira] [Updated] (FLINK-28172) Scatter dstl files into separate directories by job id

["Roman Khachatryan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-28172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Roman Khachatryan updated FLINK-28172:
--------------------------------------
        Parent: FLINK-25842
    Issue Type: Sub-task  (was: Improvement)

> Scatter dstl files into separate directories by job id
> ------------------------------------------------------
>
>                 Key: FLINK-28172
>                 URL: https://issues.apache.org/jira/browse/FLINK-28172
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / State Backends
>    Affects Versions: 1.15.0
>            Reporter: Feifan Wang
>            Assignee: Feifan Wang
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.16.0
>
>
> In the current implementation of {_}FsStateChangelogStorage{_}, dstl files from all jobs are put into the same directory (configured via {_}dstl.dfs.base-path{_}). Everything is fine if it's a filesystem like S3.But if it is a file system like hadoop, there will be some problems.
> First, there may be an upper limit to the number of files in a single directory. Increasing this threshold will greatly reduce the performance of the distributed file system.
> Second, dstl file management becomes difficult because the user cannot tell which job the dstl file belongs to, especially when the retained checkpoint is turned on.
> h3. Propose
>  # create a subdirectory named with the job id under the _dstl.dfs.base-path_ directory when the job starts
>  # all dstl files upload to the subdirectory
> ( Going a step further, we can even create two levels of subdirectories under the _dstl.dfs.base-path_ directory, like _base-path/\{jobId}/dstl ._ This way, if the user configures the same dstl.dfs.base-path as state.checkpoints.dir, all files needed for job recovery will be in the same directory and well organized. )



--
This message was sent by Atlassian Jira
(v8.20.10#820010)