[jira] [Created] (FLINK-22017) Regions may never be scheduled when there are cross-region blocking edges

["Zhilong Hong (Jira)" <ji...@apache.org>]

Zhilong Hong created FLINK-22017:
------------------------------------

             Summary: Regions may never be scheduled when there are cross-region blocking edges
                 Key: FLINK-22017
                 URL: https://issues.apache.org/jira/browse/FLINK-22017
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Coordination
    Affects Versions: 1.13.0
            Reporter: Zhilong Hong
         Attachments: Illustration.jpg

For the topology with cross-region blocking edges, there are regions that may never be scheduled. The case is illustrated in the figure below.

!Illustration.jpg!

Let's denote the vertices with layer_number. It's clear that the edge connects v2_2 and v3_2 crosses region 1 and region 2. Since region 1 has no blocking edges connected to other regions, it will be scheduled first. When vertex2_2 is finished, PipelinedRegionSchedulingStrategy will trigger {{onExecutionStateChange}} for it.

As expected, region 2 will be scheduled since all its consumer partitions are consumable. But in fact region 2 won't be scheduled, because the result partition of vertex2_2 is not tagged as consumable. Whether it is consumable or not is determined by its IntermediateDataSet.

However, an IntermediateDataSet is consumable if and only if all the producers of its IntermediateResultPartitions are finished. This IntermediateDataSet will never be consumable since vertex2_3 is not scheduled. All in all, this forms a deadlock that a region will never be scheduled because it's not scheduled.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)