[jira] [Comment Edited] (FLINK-16001) Avoid using Java Streams in construction of ExecutionGraph

["Jiayi Liao (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-16001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034538#comment-17034538 ] 

Jiayi Liao edited comment on FLINK-16001 at 2/11/20 3:14 PM:
-------------------------------------------------------------

[~gjy] Thanks for your attention. Currently we're trying to optimize the job with topology of A(p=8000) -> B(p=8000), ALL_TO_ALL pattern, which takes JM nearly a minute to finish function {{attachJobGraph}}. Part of the cost is from duplicate calculating of {{PipelinedRegionComputeUtil.computePipelinedRegions}} in {{RegionPartitionReleaseStrategy}} and {{AdaptedRestartPipelinedRegionStrategyNG}}, which I can open another issue if you think the improvement is needed.


was (Author: wind_ljy):
[~gjy] Thanks for your attention. Currently we're trying to optimize the job with topology of A(p=8000) -> B(p=8000), ALL_TO_ALL pattern, which takes us nearly a minute to finish function {{attachJobGraph}}. Part of the cost is from duplicate calculating of {{PipelinedRegionComputeUtil.computePipelinedRegions}} in {{RegionPartitionReleaseStrategy}} and {{AdaptedRestartPipelinedRegionStrategyNG}}, which I can open another issue if you think the improvement is needed.

> Avoid using Java Streams in construction of ExecutionGraph
> ----------------------------------------------------------
>
>                 Key: FLINK-16001
>                 URL: https://issues.apache.org/jira/browse/FLINK-16001
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Coordination
>    Affects Versions: 1.10.0
>            Reporter: Jiayi Liao
>            Priority: Major
>
> I think we should avoid {{Java Streams}} in construction of {{ExecutionGraph}} like function {{toPipelinedRegionsSet}} in {{PipelinedRegionComputeUtil}} because the job submission is definitely performance sensitive, especially when {{distinctRegions}} has a large cardinality.
> Also includes some other places in package {{org.apache.flink.runtime.executiongraph}}
> cc [~trohrmann] [~gjy] [~zhuzh] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)