[jira] [Commented] (FLINK-26803) Merge small ChannelState file for Unaligned Checkpoint

["Dong Lin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-26803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729067#comment-17729067 ] 

Dong Lin commented on FLINK-26803:
----------------------------------

The PR of this JIRA introduced a new public evolving API CheckpointConfig@getMaxSubtasksPerChannelStateFile and public config execution.checkpointing.unaligned.max-subtasks-per-channel-state-file. I could not find the FLIP associated with this JIRA.

[~fanrui] [~pnowojski] Is there a FLIP for this change?

> Merge small ChannelState file for Unaligned Checkpoint
> ------------------------------------------------------
>
>                 Key: FLINK-26803
>                 URL: https://issues.apache.org/jira/browse/FLINK-26803
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Checkpointing, Runtime / Network
>            Reporter: Rui Fan
>            Assignee: Rui Fan
>            Priority: Major
>              Labels: pull-request-available, stale-assigned
>             Fix For: 1.17.0
>
>         Attachments: image-2022-05-05-12-36-09-969.png
>
>
> When making an unaligned checkpoint, the number of ChannelState files is TaskNumber * subtaskNumber. For high parallelism job, it writes too many small files. It causes high load for hdfs NN.
>  
> In our production, a job writes more than 50K small files for each Unaligned Checkpoint. Could we merge these files before write FileSystem? We can configure the maximum number of files each TM can write in a single Unaligned Checkpoint.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)