[jira] [Closed] (FLINK-21915) Optimize Execution#finishPartitionsAndUpdateConsumers

["Zhilong Hong (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-21915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhilong Hong closed FLINK-21915.
--------------------------------
    Resolution: Later

Due to FLINK-22017, now blocking partitions will be individually consumable once it's finished. {{finishPartitionAndUpdateConsumer}} will be called every partition is finished. The proposed optimization is no longer valid. Furthermore, this function is called only when there is intra-region edges in the graph. In this case the downstream vertices are DEPLOYING/RUNNING when the upstream vertices are FINISHED. The scenario is rare. Thus for now we just close it. If there's a new idea about it, we'd like to reopen it.

> Optimize Execution#finishPartitionsAndUpdateConsumers
> -----------------------------------------------------
>
>                 Key: FLINK-21915
>                 URL: https://issues.apache.org/jira/browse/FLINK-21915
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Coordination
>    Affects Versions: 1.13.0
>            Reporter: Zhilong Hong
>            Priority: Major
>              Labels: auto-unassigned, pull-request-available
>
> Based on the scheduler benchmark {{PartitionReleaseInBatchJobBenchmark}} introduced in FLINK-20612, we find that there's another procedure that has O(N^2) computation complexity: {{Execution#finishPartitionsAndUpdateConsumers}}. 
> Once an execution is finished, it will finish all its BLOCKING partitions and update the partition info to all consumer vertices. The procedure can be illustrated as the following pseudo code:
> {code:java}
> for all Execution in ExecutionGraph:
>   for all produced IntermediateResultPartition of the Execution:
>     for all consumer ExecutionVertex of the IntermediateResultPartition:
>       update or cache partition info{code}
> This procedure has O(N^2) complexity in total.
> Based on FLINK-21326, the consumed partitions are grouped if they are connected to the same consumer vertices. Therefore, we can update partition info of the entire ConsumedPartitionGroup in batch, rather than one by one. This will decrease the complexity from O(N^2) to O(N).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)