You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zhuoluo Yang (JIRA)" <ji...@apache.org> on 2017/08/02 11:53:00 UTC
[jira] [Updated] (FLINK-6309) Memory consumer weights should be
calculated in job vertex level
[ https://issues.apache.org/jira/browse/FLINK-6309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhuoluo Yang updated FLINK-6309:
--------------------------------
Description:
Currently, in {{PlanFinalizer}}, we travel all the job vertices to calculate the consumer weights of the memory, and then assign the weights for each job vertex. In case of a large job graph, e.g. with multiple joins, group reduces, the value of consumer weights will be very high and the available memory for each job vertex will be very low.
I think it makes more sense to calculate the consumer weights of the memory at the job vertex level (after chaining), in order to maximize the usage ratio of the memory.
was:
Currently in {{PlanFinalizer}}, we travel all the job vertices to calculate the consumer weights of the memory, and then assign the weights for each job vertex. In case of a large job graph, e.g. with multiple joins, group reduces, the value of consumer weights will be very high and the available memory for each job vertex will be very low.
I think it makes more sense to calculate the consumer weights of the memory at the job vertex level (after chaining), in order to maximize the usage ratio of the memory.
> Memory consumer weights should be calculated in job vertex level
> ----------------------------------------------------------------
>
> Key: FLINK-6309
> URL: https://issues.apache.org/jira/browse/FLINK-6309
> Project: Flink
> Issue Type: Improvement
> Components: Optimizer
> Reporter: Kurt Young
> Assignee: Xu Pingyong
>
> Currently, in {{PlanFinalizer}}, we travel all the job vertices to calculate the consumer weights of the memory, and then assign the weights for each job vertex. In case of a large job graph, e.g. with multiple joins, group reduces, the value of consumer weights will be very high and the available memory for each job vertex will be very low.
> I think it makes more sense to calculate the consumer weights of the memory at the job vertex level (after chaining), in order to maximize the usage ratio of the memory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)