You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zhu Zhu (Jira)" <ji...@apache.org> on 2019/12/19 09:39:00 UTC
[jira] [Updated] (FLINK-15325) Input location preference which
affects task distribution may make certain job performance worse
[ https://issues.apache.org/jira/browse/FLINK-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhu Zhu updated FLINK-15325:
----------------------------
Description:
When running TPC-DS jobs in a session cluster, we observed that sometimes tasks are not evenly distributed in TMs. The root cause turned out to be that the downstream tasks tend to be TM or host local with its input tasks. This helps to reduce network shuffle.
However, in certain cases, like the topology presented in the attached image, jamming the input task's TM and machine with downstream tasks would affect the performance. In this case, respecting input location preferences is causing troubles more than bringing benefits.
So I'm wondering whether we should introduce a config to disable input location preference?
> Input location preference which affects task distribution may make certain job performance worse
> -------------------------------------------------------------------------------------------------
>
> Key: FLINK-15325
> URL: https://issues.apache.org/jira/browse/FLINK-15325
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.10.0
> Reporter: Zhu Zhu
> Priority: Major
> Attachments: D58ADB03-7187-46B1-B077-91E5005FD463.png
>
>
> When running TPC-DS jobs in a session cluster, we observed that sometimes tasks are not evenly distributed in TMs. The root cause turned out to be that the downstream tasks tend to be TM or host local with its input tasks. This helps to reduce network shuffle.
> However, in certain cases, like the topology presented in the attached image, jamming the input task's TM and machine with downstream tasks would affect the performance. In this case, respecting input location preferences is causing troubles more than bringing benefits.
> So I'm wondering whether we should introduce a config to disable input location preference?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)