You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yu Li (Jira)" <ji...@apache.org> on 2020/05/01 12:39:00 UTC
[jira] [Updated] (FLINK-17073) Slow checkpoint cleanup causing OOMs
[ https://issues.apache.org/jira/browse/FLINK-17073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yu Li updated FLINK-17073:
--------------------------
Fix Version/s: (was: 1.10.1)
1.10.2
> Slow checkpoint cleanup causing OOMs
> ------------------------------------
>
> Key: FLINK-17073
> URL: https://issues.apache.org/jira/browse/FLINK-17073
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.7.3, 1.8.0, 1.9.0, 1.10.0, 1.11.0
> Reporter: Till Rohrmann
> Priority: Critical
> Fix For: 1.11.0, 1.10.2, 1.9.4
>
>
> A user reported that he sees a decline in checkpoint cleanup speed when upgrading from Flink 1.7.2 to 1.10.0. The result is that a lot of cleanup tasks are waiting in the execution queue occupying memory. Ultimately, the JM process dies with an OOM.
> Compared to Flink 1.7.2, we introduced a dedicated {{ioExecutor}} which is used by the {{HighAvailabilityServices}} (FLINK-11851). Before, we use the {{AkkaRpcService}} thread pool which was a {{ForkJoinPool}} with a max parallelism of 64. Now it is a {{FixedThreadPool}} with as many threads as CPU cores. This change might have caused the decline in completed checkpoint discard throughput. This suspicion needs to be validated before trying to fix it!
> [1] https://lists.apache.org/thread.html/r390e5d775878918edca0b6c9f18de96f828c266a888e34ed30ce8494%40%3Cuser.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)