[GitHub] [flink] GJL commented on a change in pull request #11691: [FLINK-14234][runtime] Notifies all kinds of consumable partitions to SchedulingStrategy

[GitBox <gi...@apache.org>]

GJL commented on a change in pull request #11691: [FLINK-14234][runtime] Notifies all kinds of consumable partitions to SchedulingStrategy
URL: https://github.com/apache/flink/pull/11691#discussion_r406411129
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java
 ##########
 @@ -527,6 +531,26 @@ private boolean isNotifiable(
 		return false;
 	}
 
+	private void maybeNotifyBlockingPartitionsConsumable(final ExecutionVertexID executionVertexId) {
+		final ExecutionVertex executionVertex = getExecutionVertex(executionVertexId);
+
+		if (executionVertex.getExecutionState() != ExecutionState.FINISHED) {
+			return;
+		}
+
+		final Set<IntermediateResultPartitionID> consumableResultPartitions = new HashSet<>();
+		for (IntermediateResultPartition resultPartition : executionVertex.getProducedPartitions().values()) {
+			if (resultPartition.getResultType().isBlocking() && resultPartition.isConsumable()) {
 
 Review comment:
   We currently define an `IntermediateResultPartition` to be consumable if all of of the partitions in the intermediate result are consumable. I think one of the reasons `LazyFromSourcesSchedulingStrategy` is so bloated, is that we wanted to enable the `SchedulingStrategy` to define consumability. That is, a `SchedulingStrategy` could decide to schedule downstream operators as soon as one partition is finished (instead of waiting for the entire intermediate result). Now my questions are:
   1. Is it already possible to implement the above requirement in Flink, or does something impede us from consuming partitions of an incomplete intermediate result?
   1. How likely is it that we have to implement the above requirement? If we merge this PR and later change the contract of `notifyPartitionConsumable()`, `LazyFromSourcesSchedulingStrategy` will break. Can we postpone the decision whether to simplify `LazyFromSourcesSchedulingStrategy`?
   1. What should be the behavior for Pipelined Region Scheduling?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services