You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "fanrui (Jira)" <ji...@apache.org> on 2022/06/10 08:54:00 UTC

[jira] [Commented] (FLINK-26803) Merge small ChannelState file for Unaligned Checkpoint

    [ https://issues.apache.org/jira/browse/FLINK-26803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17552631#comment-17552631 ] 

fanrui commented on FLINK-26803:
--------------------------------

Hi everyone, I've done a POC for merging the channel state files and organized the documentation. Hope the masters can participate in the discussion in the doc or JIRA.

[https://docs.google.com/document/d/1uGp1gJxxcJMUCh5DxzPklGc2SicMN-Dk-YOce2XMaAg/edit?usp=sharing]

POC branch: [https://github.com/1996fanrui/flink/commits/26803/merge-channel-file-POC]

The latest commit: [https://github.com/1996fanrui/flink/commit/48780677683dad43cfb2211781493a234bab087c]

> Merge small ChannelState file for Unaligned Checkpoint
> ------------------------------------------------------
>
>                 Key: FLINK-26803
>                 URL: https://issues.apache.org/jira/browse/FLINK-26803
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Checkpointing, Runtime / Network
>            Reporter: fanrui
>            Priority: Major
>         Attachments: image-2022-05-05-12-36-09-969.png
>
>
> When making an unaligned checkpoint, the number of ChannelState files is TaskNumber * subtaskNumber. For high parallelism job, it writes too many small files. It causes high load for hdfs NN.
>  
> In our production, a job writes more than 50K small files for each Unaligned Checkpoint. Could we merge these files before write FileSystem? We can configure the maximum number of files each TM can write in a single Unaligned Checkpoint.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)