You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Anton Ryabtsev (Jira)" <ji...@apache.org> on 2019/09/03 11:24:00 UTC
[jira] [Commented] (FLINK-12122) Spread out tasks evenly across all
available registered TaskManagers
[ https://issues.apache.org/jira/browse/FLINK-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921347#comment-16921347 ]
Anton Ryabtsev commented on FLINK-12122:
----------------------------------------
[~NicoK], the issue with the setting the same parallelism for all tasks is that you run into another memory issue: as you said it requires additional native memory for thread stack, network buffers and it increases the load to GC.
It hits my project very badly as I have a coefficient between max(slots per task)/min(slots per task) as 5 and my Flink's cluster is running under native memory limit per process.
> Spread out tasks evenly across all available registered TaskManagers
> --------------------------------------------------------------------
>
> Key: FLINK-12122
> URL: https://issues.apache.org/jira/browse/FLINK-12122
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Affects Versions: 1.6.4, 1.7.2, 1.8.0
> Reporter: Till Rohrmann
> Priority: Major
> Attachments: image-2019-05-21-12-28-29-538.png, image-2019-05-21-13-02-50-251.png
>
>
> With Flip-6, we changed the default behaviour how slots are assigned to {{TaskManages}}. Instead of evenly spreading it out over all registered {{TaskManagers}}, we randomly pick slots from {{TaskManagers}} with a tendency to first fill up a TM before using another one. This is a regression wrt the pre Flip-6 code.
> I suggest to change the behaviour so that we try to evenly distribute slots across all available {{TaskManagers}} by considering how many of their slots are already allocated.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)