[jira] [Commented] (FLINK-27155) Reduce multiple reads to the same Changelog file in the same taskmanager during restore

["Roman Khachatryan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-27155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558041#comment-17558041 ] 

Roman Khachatryan commented on FLINK-27155:
-------------------------------------------

Many thanks for writing the design doc, [~Feifan Wang].
In general, the described approach makes a lot of sense to me.

I wanted to add these comments (I've sent a request to get commenter access):
2.3: We could probably re-use common thread pool, such as RuntimeEnvironment.asyncOperationsThreadPool
2.4: Could you explain why StateChangeFormat methods have to be static?
2.4: Just to clarify: the new "cache" component will essentially customize two lines from StateChangeFormat:
    FSDataInputStream stream = handle.openInputStream();
    DataInputViewStreamWrapper input = wrap(stream);
    Right?
    So after some refactoring we can have a non-caching version and then add a caching one. 

2.5.3 / 2.5.4: I'm not sure those approaches can handle pipelined regions properly and without further complication

> Reduce multiple reads to the same Changelog file in the same taskmanager during restore
> ---------------------------------------------------------------------------------------
>
>                 Key: FLINK-27155
>                 URL: https://issues.apache.org/jira/browse/FLINK-27155
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / State Backends
>            Reporter: Feifan Wang
>            Assignee: Feifan Wang
>            Priority: Major
>             Fix For: 1.16.0
>
>
> h3. Background
> In the current implementation, State changes of different operators in the same taskmanager may be written to the same changelog file, which effectively reduces the number of files and requests to DFS.
> But on the other hand, the current implementation also reads the same changelog file multiple times on recovery. More specifically, the number of times the same changelog file is accessed is related to the number of ChangeSets contained in it. And since each read needs to skip the preceding bytes, this network traffic is also wasted.
> The result is a lot of unnecessary request to DFS when there are multiple slots and keyed state in the same taskmanager.
> h3. Proposal
> We can reduce multiple reads to the same changelog file in the same taskmanager during restore.
> One possible approach is to read the changelog file all at once and cache it in memory or local file for a period of time when reading the changelog file.
> I think this could be a subtask of [v2 FLIP-158: Generalized incremental checkpoints|https://issues.apache.org/jira/browse/FLINK-25842] .
> Hi [~ym] , [~roman]  how do you think about ?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)