You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "fanrui (Jira)" <ji...@apache.org> on 2022/05/05 02:08:00 UTC
[jira] [Comment Edited] (FLINK-26803) Merge small ChannelState file for Unaligned Checkpoint
[ https://issues.apache.org/jira/browse/FLINK-26803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531603#comment-17531603 ]
fanrui edited comment on FLINK-26803 at 5/5/22 2:07 AM:
--------------------------------------------------------
Hi [~pnowojski] Thanks for your feedback.
I'm sorry. I didn't express my thoughts clearly. Let me add some information.
For this job, it's a 30% reduction. But most Flink jobs do not have very large state, and usually, their number of files is not very large. If the backpressure is severe and UC is enabled, the number of files for these jobs will become high.
{*}The idea I want to express is{*}: _For a flink job with a small state, high parallelism and a large number of tasks, if ChannelStateFile is optimized, the number of files may be reduced by more than 80% or even 90%. And I believe production small state jobs are more._
When the backpressure is not severe, hdfs can run normally, if a large number of jobs have severe back pressure at the same time, it may bring several times the pressure to hdfs. The file number mechanism of UC makes it difficult for Flink SRE to estimate the hdfs capacity.
was (Author: fanrui):
Hi [~pnowojski] Thanks for your feedback.
I'm sorry. I didn't express my thoughts clearly. Let me add some information.
For this job, it's a 30% reduction. But most Flink jobs do not have very large state, and usually, their number of files is not very large. If the backpressure is severe and UC is enabled, the number of files for these jobs will become high.
{*}The idea I want to express is{*}: _For a flink job with a small state, high parallelism and a large number of jobs, if ChannelStateFile is optimized, the number of files may be reduced by more than 80%. And I believe production small state jobs are more._
When the backpressure is not severe, hdfs can run normally, if a large number of jobs have severe back pressure at the same time, it may bring several times the pressure to hdfs. The file number mechanism of UC makes it difficult for Flink SRE to estimate the hdfs capacity.
> Merge small ChannelState file for Unaligned Checkpoint
> ------------------------------------------------------
>
> Key: FLINK-26803
> URL: https://issues.apache.org/jira/browse/FLINK-26803
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / Network
> Reporter: fanrui
> Priority: Major
>
> When making an unaligned checkpoint, the number of ChannelState files is TaskNumber * subtaskNumber. For high parallelism job, it writes too many small files. It causes high load for hdfs NN.
>
> In our production, a job writes more than 50K small files for each Unaligned Checkpoint. Could we merge these files before write FileSystem? We can configure the maximum number of files each TM can write in a single Unaligned Checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)