You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Till Rohrmann (JIRA)" <ji...@apache.org> on 2019/07/10 07:31:00 UTC
[jira] [Updated] (FLINK-12138) Limit input split count of each
source task for better failover experience
[ https://issues.apache.org/jira/browse/FLINK-12138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Till Rohrmann updated FLINK-12138:
----------------------------------
Component/s: (was: Runtime / Coordination)
Runtime / Task
> Limit input split count of each source task for better failover experience
> --------------------------------------------------------------------------
>
> Key: FLINK-12138
> URL: https://issues.apache.org/jira/browse/FLINK-12138
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Affects Versions: 1.9.0
> Reporter: Zhu Zhu
> Assignee: Zhu Zhu
> Priority: Major
>
> Flink currently use an InputSplitAssigner to dynamically assign input splits to source tasks. A task requests a new split after finishes processing the previous one. Thus to achieve a better load balance.
> However, in cases that the slots are fewer than the source tasks, only the first launched source tasks can request splits and it will last till all the splits are consumed. This is not failover friendly, as users sometimes intentionally set a larger parallelism to reduce the failover impact.
> For example, a job runs in an 10 slots session and it has an 1000 parallelism source vertex to consume 10000 splits, all vertices are not connected to others. Currently, 10 of 1000 source task will be launched and will only finish after all the input splits are consumed. If a task fails, at most ~1000 splits need to be re-processed. While if 1000 tasks can run at once, only ~10 splits needs to be re-processed.
>
> We's propose add a cap for the input splits count that each source task shall process. Once the cap is reached, the task cannot get any more split from the InputSplitAssigner and finishes then. Thus slot space can be made for other source tasks.
> Theoretically, it would be proper to set the cap to be max(Input split size)/avg(input split size).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)