[jira] [Resolved] (FLINK-11869) [checkpoint] Make buffer size in checkpoint stream factory configurable

["Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-11869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tzu-Li (Gordon) Tai resolved FLINK-11869.
-----------------------------------------
    Resolution: Fixed

Merged for 1.9.0: bc5e8a77aa3419afc03c0751dd339e5027cf3664

> [checkpoint] Make buffer size in checkpoint stream factory configurable
> -----------------------------------------------------------------------
>
>                 Key: FLINK-11869
>                 URL: https://issues.apache.org/jira/browse/FLINK-11869
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Checkpointing
>            Reporter: Yun Tang
>            Assignee: Yun Tang
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.9.0
>
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, the default buffer size for {{FsCheckpointStateOutputStream}} is only 4KB. This would case a lot of IOPS if stream is large. Unfortunately, when user want to checkpoint on a totally disaggregated file system which has no data node manager running in local machine, they might have a IOPS limit or cannot serve too many IOPS at a time. This would cause the checkpoint duration really large and might expire often. 
> If we want to increase this buffer size, we have to increase the {{fileStateThreshold}} to indirectly increase the buffer size. However, as we all know, too many not-so-small {{ByteStreamStateHandle}} returned to checkpoint coordinator would easily cause job manager OOM and checkpoint meta file large.
> We should also make the buffer size configurable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)