You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Jelmer Kuperus (JIRA)" <ji...@apache.org> on 2018/05/04 15:14:00 UTC
[jira] [Updated] (FLINK-9297) Make size of JobManager's ioExecutor
configurable
[ https://issues.apache.org/jira/browse/FLINK-9297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jelmer Kuperus updated FLINK-9297:
----------------------------------
Description:
With flink 1.4.0 cleaning of checkpoints is handled on the job manager, the SharedStateRegistry uses the job manager's [io executor|https://github.com/apache/flink/blob/e5ed2fbc4a31808bea8be89b371bbe4269288003/flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala#L1999] as the executor to schedule disposals of checkpoint state on. This executor has a fixed size equal to the number of cpu cores. When a lot of small files are created by many jobs and a slow filesystem is used this may not be enough.
It would be good to make this setting configurable. Initializing an executor service with the number of cpu's on the system makes sense for cpu intensive tasks less so for io based workloads
was:
With flink 1.4.0 cleaning of checkpoints is handled on the job manager, the SharedStateRegistry uses the job manager's [io executor|https://github.com/apache/flink/blob/e5ed2fbc4a31808bea8be89b371bbe4269288003/flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala#L1999] as the executor to schedule disposals of checkpoint state on. This executor has a fixed size equal to the number of cpu cores. When a lot of small files are created this may not be enough.
It would be good to make this setting configurable. Initializing an executor service with the number of cpu's on the system makes sense for cpu intensive tasks less so for io based workloads
> Make size of JobManager's ioExecutor configurable
> -------------------------------------------------
>
> Key: FLINK-9297
> URL: https://issues.apache.org/jira/browse/FLINK-9297
> Project: Flink
> Issue Type: Improvement
> Components: JobManager
> Affects Versions: 1.4.0, 1.4.1, 1.4.2
> Reporter: Jelmer Kuperus
> Priority: Major
>
> With flink 1.4.0 cleaning of checkpoints is handled on the job manager, the SharedStateRegistry uses the job manager's [io executor|https://github.com/apache/flink/blob/e5ed2fbc4a31808bea8be89b371bbe4269288003/flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala#L1999] as the executor to schedule disposals of checkpoint state on. This executor has a fixed size equal to the number of cpu cores. When a lot of small files are created by many jobs and a slow filesystem is used this may not be enough.
> It would be good to make this setting configurable. Initializing an executor service with the number of cpu's on the system makes sense for cpu intensive tasks less so for io based workloads
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)