[jira] [Created] (FLINK-21326) Optimize building topology when initializing ExecutionGraph

["Zhilong Hong (Jira)" <ji...@apache.org>]

Zhilong Hong created FLINK-21326:
------------------------------------

             Summary: Optimize building topology when initializing ExecutionGraph
                 Key: FLINK-21326
                 URL: https://issues.apache.org/jira/browse/FLINK-21326
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Coordination
            Reporter: Zhilong Hong
             Fix For: 1.13.0


The main idea of optimizing the procedure of building topology is to put all the vertices that consumed the same result partitions into one group, and put all the result partitions that have the same consumer vertices into one consumer group. The corresponding data structure is {{ConsumedPartitionGroup}} and {{ConsumerVertexGroup}}. {{EdgeManager}} is used to store the relationship between the groups. The procedure of creating {{ExecutionEdge}} is replaced with building {{EdgeManager}}.

With these improvements, the complexity of building topology in ExecutionGraph decreases from O(N^2) to O(N). 

Furthermore, {{ExecutionEdge}} and all its related calls are replaced with {{ConsumedPartitionGroup}} and {{ConsumerVertexGroup}}.

The detailed design doc is located at: https://docs.google.com/document/d/1OjGAyJ9Z6KsxcMtBHr6vbbrwP9xye7CdCtrLvf8dFYw/edit?usp=sharing



--
This message was sent by Atlassian Jira
(v8.3.4#803005)