[jira] [Created] (FLINK-21332) Optimize releasing result partitions in RegionPartitionReleaseStrategy

["Zhilong Hong (Jira)" <ji...@apache.org>]

Zhilong Hong created FLINK-21332:
------------------------------------

             Summary: Optimize releasing result partitions in RegionPartitionReleaseStrategy
                 Key: FLINK-21332
                 URL: https://issues.apache.org/jira/browse/FLINK-21332
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Coordination
            Reporter: Zhilong Hong
             Fix For: 1.13.0


RegionPartitionReleaseStrategy is responsible for releasing result partitions when all the downstream tasks finish.

The current implementation is:
{code:java}
for each consumed SchedulingResultPartition of current finished SchedulingPipelinedRegion:
  for each consumer SchedulingPipelinedRegion of the SchedulingResultPartition:
    if all the regions are finished:
      release the partitions
{code}
The time complexity of releasing a result partition is O(N^2). However, considering that during the entire stage, all the result partitions need to be released, the time complexity is actually O(N^3).

After the optimization of DefaultSchedulingTopology, the consumed result partitions are grouped. Since the result partitions in one group are isomorphic, we can just cache the finished status of result partition groups and the corresponding pipeline regions.

The optimized implementation is:
{code:java}
for each ConsumedPartitionGroup of current finished SchedulingPipelinedRegion:
  if all consumer SchedulingPipelinedRegion of the ConsumedPartitionGroup are finished:
    set the ConsumePartitionGroup to be fully consumed
    for result partition in the ConsumePartitionGroup:
      if all the ConsumePartitionGroups it belongs to are fully consumed:
        release the result partition
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)