You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/12/26 10:13:52 UTC

[GitHub] [flink] reswqa opened a new pull request, #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

reswqa opened a new pull request, #21560:
URL: https://github.com/apache/flink/pull/21560

   ## What is the purpose of the change
   
   *At present, downstream task can be scheduled only if all upstream tasks finished in adaptive batch scheduler. We can do some improvements to support schedule downstream tasks when the upstream's some subtask is not completely finished. After the upstream task's data is totally produced, we need to update the result partition's information of this subtask to trigger downstream task's consumption.*
   
   ## Brief change log
   
     - *Introduce PartialFinishedInputConsumableDecider.*
     - *Serialize and cache shuffle descriptor one by one instead of group.*
     - *UnknownInputChannel supports update resultPartitionId when transform to local input channel.*
     - *Introduce ResultPartitionType#isHybridResultPartition.*
     - *Adaptive batch scheduler supports schedule downstream tasks when partial producer finished.*
   
   
   ## Verifying this change
   
   This change added unit tests.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1060429247


##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/InputGateDeploymentDescriptor.java:
##########
@@ -75,7 +77,7 @@ public class InputGateDeploymentDescriptor implements Serializable {
     private transient ShuffleDescriptor[] inputChannels;
 
     /** Serialized value of shuffle descriptors. */
-    private MaybeOffloaded<ShuffleDescriptor[]> serializedInputChannels;
+    private final List<MaybeOffloaded<ShuffleDescriptor>> serializedInputChannels;

Review Comment:
   This is indeed a reasonable concerns. I changed this logical to serialize `shuffleDescriptors` in batch, and the `IGDD(InputGateDeploymentDescriptor)` stores `List<MaybeOffloaded<ShuffleDescriptorAndIndex[]>>`, we can calculates all producer's `shuffleDescriptor` from this list.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] xintongsong commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
xintongsong commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1058108777


##########
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/InternalExecutionGraphAccessor.java:
##########
@@ -122,4 +122,6 @@ List<ShuffleDescriptor> getClusterPartitionShuffleDescriptors(
             IntermediateDataSetID intermediateResultPartition);
 
     MarkPartitionFinishedStrategy getMarkPartitionFinishedStrategy();
+
+    boolean isHybridEnableConsumePartialFinishedProducer();

Review Comment:
   I'm not entirely sure about introducing a hybrid specific interface in execution graph. Haven't think of an alternative way though.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/PartialFinishedInputConsumableDecider.java:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.strategy;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+
+/**
+ * {@link PartialFinishedInputConsumableDecider} is a special {@link InputConsumableDecider}. The
+ * input is considered to be consumable:
+ *
+ * <ul>
+ *   <li>for hybrid input: when partial producer partitions are finished.
+ *   <li>for blocking input: when all producer partitions are finished.
+ * </ul>
+ */
+public class PartialFinishedInputConsumableDecider implements InputConsumableDecider {
+    public static final int NUM_FINISHED_PARTITIONS_AS_CONSUMABLE = 1;
+
+    @Override
+    public boolean isInputConsumable(
+            SchedulingExecutionVertex executionVertex,
+            Set<ExecutionVertexID> verticesToDeploy,
+            Map<ConsumedPartitionGroup, Boolean> consumableStatusCache) {
+        for (ConsumedPartitionGroup consumedPartitionGroup :
+                executionVertex.getConsumedPartitionGroups()) {
+
+            if (!consumableStatusCache.computeIfAbsent(
+                    consumedPartitionGroup, this::isConsumedPartitionGroupConsumable)) {
+                return false;
+            }

Review Comment:
   When there're multiple hybrid partition groups, shall we require all groups to have at least one finished partition? Or one finished partition from any of the groups?



##########
flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java:
##########
@@ -670,6 +670,24 @@ public enum SchedulerType {
                                             code(SPECULATIVE_ENABLED.key()))
                                     .build());
 
+    @Documentation.Section({
+        Documentation.Sections.EXPERT_SCHEDULING,
+        Documentation.Sections.ALL_JOB_MANAGER
+    })
+    public static final ConfigOption<Boolean> CONSUME_PARTIAL_FINISHED_PRODUCER_ENABLED =

Review Comment:
   The two configs (`ONLY_CONSUME_FINISHED_PARTITION` and `CONSUME_PARTIAL_FINISHED_PRODUCER_ENABLED`) are quite alike, which may confuse users.
   - It's hard to understand the differences between them.
   - There could be conflicts. E.g., allow consuming unfinished partitions, while not allow consuming partition finished producer.
   
   I wonder if we can combine them into one config, which takes a enum type of supported values.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/CachedShuffleDescriptors.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.deployment;
+
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor.MaybeOffloaded;
+import org.apache.flink.runtime.executiongraph.IntermediateResultPartition;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.shuffle.ShuffleDescriptor;
+import org.apache.flink.util.function.FunctionWithException;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** {@link ShuffleDescriptor}s cache for a {@link ConsumedPartitionGroup} */
+public class CachedShuffleDescriptors {
+    /**
+     * Stores all serialized shuffle descriptors. For unknown shuffle descriptor, it will be
+     * replaced by real shuffle descriptor after upstream task finished.
+     */
+    private final List<MaybeOffloaded<ShuffleDescriptor>> serializedShuffleDescriptors;
+
+    /**
+     * Stores all to be serialized shuffle descriptors, They will be serialized and replace
+     * corresponding value(unknown shuffle descriptor) in serializedShuffleDescriptors during the
+     * next time TDD is generated.
+     */
+    private final Map<ShuffleDescriptor, Integer> toBeSerialized;

Review Comment:
   And why not eagerly serialize the descriptor and replace the previous unknown descriptor in  `markPartitionFinished`?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/CachedShuffleDescriptors.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.deployment;
+
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor.MaybeOffloaded;
+import org.apache.flink.runtime.executiongraph.IntermediateResultPartition;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.shuffle.ShuffleDescriptor;
+import org.apache.flink.util.function.FunctionWithException;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** {@link ShuffleDescriptor}s cache for a {@link ConsumedPartitionGroup} */
+public class CachedShuffleDescriptors {
+    /**
+     * Stores all serialized shuffle descriptors. For unknown shuffle descriptor, it will be
+     * replaced by real shuffle descriptor after upstream task finished.
+     */
+    private final List<MaybeOffloaded<ShuffleDescriptor>> serializedShuffleDescriptors;
+
+    /**
+     * Stores all to be serialized shuffle descriptors, They will be serialized and replace
+     * corresponding value(unknown shuffle descriptor) in serializedShuffleDescriptors during the
+     * next time TDD is generated.
+     */
+    private final Map<ShuffleDescriptor, Integer> toBeSerialized;

Review Comment:
   Why do we need a map? Would it be better to use a queue of tuples?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/InputGateDeploymentDescriptor.java:
##########
@@ -75,7 +77,7 @@ public class InputGateDeploymentDescriptor implements Serializable {
     private transient ShuffleDescriptor[] inputChannels;
 
     /** Serialized value of shuffle descriptors. */
-    private MaybeOffloaded<ShuffleDescriptor[]> serializedInputChannels;
+    private final List<MaybeOffloaded<ShuffleDescriptor>> serializedInputChannels;

Review Comment:
   A serialized value will be offloaded only if it's larger than the min offloading size (default 1MB). This practically means shuffler descriptors will no longer be offloaded, because a single shuffle descriptor can hardly be larger than 1MB. Moreover, if we configure the min offloading size to a smaller value, than each shuffle descriptor will be a separate blog object, which significantly increases the load of the blob server.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java:
##########
@@ -762,7 +763,9 @@ private static PartitionInfo createPartitionInfo(
         ShuffleDescriptor shuffleDescriptor =
                 getConsumedPartitionShuffleDescriptor(
                         consumedPartition,
-                        TaskDeploymentDescriptorFactory.PartitionLocationConstraint.MUST_BE_KNOWN);
+                        TaskDeploymentDescriptorFactory.PartitionLocationConstraint.MUST_BE_KNOWN,
+                        // because partition is already finished, false is fair enough.

Review Comment:
   Why is the partition finished?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1060267262


##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/CachedShuffleDescriptors.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.deployment;
+
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor.MaybeOffloaded;
+import org.apache.flink.runtime.executiongraph.IntermediateResultPartition;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.shuffle.ShuffleDescriptor;
+import org.apache.flink.util.function.FunctionWithException;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** {@link ShuffleDescriptor}s cache for a {@link ConsumedPartitionGroup} */
+public class CachedShuffleDescriptors {
+    /**
+     * Stores all serialized shuffle descriptors. For unknown shuffle descriptor, it will be
+     * replaced by real shuffle descriptor after upstream task finished.
+     */
+    private final List<MaybeOffloaded<ShuffleDescriptor>> serializedShuffleDescriptors;
+
+    /**
+     * Stores all to be serialized shuffle descriptors, They will be serialized and replace
+     * corresponding value(unknown shuffle descriptor) in serializedShuffleDescriptors during the
+     * next time TDD is generated.
+     */
+    private final Map<ShuffleDescriptor, Integer> toBeSerialized;

Review Comment:
   > Why do we need a map? Would it be better to use a queue of tuples?
   
   Yes, queue is better than map.
   
   > And why not eagerly serialize the descriptor and replace the previous unknown descriptor in `markPartitionFinished`?
   
   Because we can not get the serializer during mark partition finished. This is also the reason why `serializeShuffleDescriptors` has param of `shuffleDescriptorSerializer`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1060265216


##########
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/PartialFinishedInputConsumableDecider.java:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.strategy;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+
+/**
+ * {@link PartialFinishedInputConsumableDecider} is a special {@link InputConsumableDecider}. The
+ * input is considered to be consumable:
+ *
+ * <ul>
+ *   <li>for hybrid input: when partial producer partitions are finished.
+ *   <li>for blocking input: when all producer partitions are finished.
+ * </ul>
+ */
+public class PartialFinishedInputConsumableDecider implements InputConsumableDecider {
+    public static final int NUM_FINISHED_PARTITIONS_AS_CONSUMABLE = 1;
+
+    @Override
+    public boolean isInputConsumable(
+            SchedulingExecutionVertex executionVertex,
+            Set<ExecutionVertexID> verticesToDeploy,
+            Map<ConsumedPartitionGroup, Boolean> consumableStatusCache) {
+        for (ConsumedPartitionGroup consumedPartitionGroup :
+                executionVertex.getConsumedPartitionGroups()) {
+
+            if (!consumableStatusCache.computeIfAbsent(
+                    consumedPartitionGroup, this::isConsumedPartitionGroupConsumable)) {
+                return false;
+            }

Review Comment:
   I think it is reasonable to require all groups to have at least one finished partition. The main reason is that in the case of multiple inputs, if the low priority input is finished first, the downstream cannot consume data even if it is scheduled in advance, which will cause a waste of resources to some extent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] flinkbot commented on pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #21560:
URL: https://github.com/apache/flink/pull/21560#issuecomment-1365064472

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "c98c28755e3f2febbb926a741006ae298653f41c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c98c28755e3f2febbb926a741006ae298653f41c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c98c28755e3f2febbb926a741006ae298653f41c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1062417178


##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/CachedShuffleDescriptors.java:
##########
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.deployment;
+
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor.MaybeOffloaded;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptorFactory.ShuffleDescriptorAndIndex;
+import org.apache.flink.runtime.executiongraph.IntermediateResultPartition;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.shuffle.ShuffleDescriptor;
+import org.apache.flink.util.function.FunctionWithException;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** {@link ShuffleDescriptor}s cache for a {@link ConsumedPartitionGroup}. */
+public class CachedShuffleDescriptors {
+    /**
+     * Stores all serialized shuffle descriptors with indexes. For a partition, it may be added a
+     * serialized unknown shuffle descriptor to this list first, and then added the real descriptor
+     * later.
+     */
+    private final List<MaybeOffloaded<ShuffleDescriptorAndIndex[]>> serializedShuffleDescriptors;
+
+    /**
+     * Stores all to be serialized shuffle descriptors, They will be serialized and added to
+     * serializedShuffleDescriptors during the next time TaskDeploymentDescriptor is generated.
+     */
+    private final Queue<ShuffleDescriptorAndIndex> toBeSerialized;
+
+    /** Stores the mapping of resultPartitionId to index subscripts in consumed partition group. */
+    private final Map<IntermediateResultPartitionID, Integer> resultPartitionIdToIndex;
+
+    public CachedShuffleDescriptors(
+            ConsumedPartitionGroup consumedPartitionGroup,
+            MaybeOffloaded<ShuffleDescriptorAndIndex[]> initialSerializedShuffleDescriptors) {

Review Comment:
   fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1060273190


##########
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java:
##########
@@ -762,7 +763,9 @@ private static PartitionInfo createPartitionInfo(
         ShuffleDescriptor shuffleDescriptor =
                 getConsumedPartitionShuffleDescriptor(
                         consumedPartition,
-                        TaskDeploymentDescriptorFactory.PartitionLocationConstraint.MUST_BE_KNOWN);
+                        TaskDeploymentDescriptorFactory.PartitionLocationConstraint.MUST_BE_KNOWN,
+                        // because partition is already finished, false is fair enough.

Review Comment:
   Because `createPartitionInfo` will only be invoked from `finishPartitionsAndUpdateConsumers`, which has been already finished this partition. But in order not to be misused later, I renamed it to `createFinishedPartitionInfo`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on PR #21560:
URL: https://github.com/apache/flink/pull/21560#issuecomment-1373058743

   @xintongsong Thanks for the review, I have updated this pr, PTAL~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1060304453


##########
flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java:
##########
@@ -670,6 +670,24 @@ public enum SchedulerType {
                                             code(SPECULATIVE_ENABLED.key()))
                                     .build());
 
+    @Documentation.Section({
+        Documentation.Sections.EXPERT_SCHEDULING,
+        Documentation.Sections.ALL_JOB_MANAGER
+    })
+    public static final ConfigOption<Boolean> CONSUME_PARTIAL_FINISHED_PRODUCER_ENABLED =

Review Comment:
   Good idea, I will use an enum type to replace this two config option.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1062442622


##########
flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java:
##########
@@ -650,24 +650,46 @@ public enum SchedulerType {
                     .withDescription(
                             "The JobManager's ResourceID. If not configured, the ResourceID will be generated randomly.");
 
+    /** Type of consume partition mode of hybrid edge. */
+    public enum HybridConsumePartitionMode {
+        ONLY_CONSUME_ALL_FINISHED(true),
+        CAN_CONSUME_PARTIAL_FINISHED(true),
+        CAN_CONSUME_UN_FINISHED(false);
+
+        private final boolean onlyConsumeFinishedPartition;
+
+        HybridConsumePartitionMode(boolean onlyConsumeFinishedPartition) {
+            this.onlyConsumeFinishedPartition = onlyConsumeFinishedPartition;
+        }
+
+        public boolean isOnlyConsumeFinishedPartition() {
+            return onlyConsumeFinishedPartition;
+        }
+    }
+
     @Documentation.Section({
         Documentation.Sections.EXPERT_SCHEDULING,
         Documentation.Sections.ALL_JOB_MANAGER
     })
-    public static final ConfigOption<Boolean> ONLY_CONSUME_FINISHED_PARTITION =
-            key("jobmanager.partition.hybrid.only-consume-finished-partition")
-                    .booleanType()
+    public static final ConfigOption<HybridConsumePartitionMode> HYBRID_CONSUME_PARTITION_MODE =
+            key("jobmanager.partition.hybrid.consume-partition-mode")
+                    .enumType(HybridConsumePartitionMode.class)
                     .noDefaultValue()
                     .withDescription(
                             Description.builder()
                                     .text(
-                                            "Controls whether the scheduler only allows downstream task consume finished partition. "
-                                                    + "Note that this option is allowed only when %s has been set to %s, "
-                                                    + "and if you also enable speculative execution(%s has been set to true),"
-                                                    + "this option can only be set to true.",
+                                            "Controls how the downstream task consumes hybrid type producer edge. "
+                                                    + "Note that this option is allowed only when %s has been set to %s. "
+                                                    + "Accepted values are:",
                                             code(SCHEDULER.key()),
-                                            code(SchedulerType.AdaptiveBatch.name()),
-                                            code(SPECULATIVE_ENABLED.key()))
+                                            code(SchedulerType.AdaptiveBatch.name()))
+                                    .list(
+                                            text(
+                                                    "'ONLY_CONSUME_ALL_FINISHED': Downstream tasks can consume data only when all producers are finished."),

Review Comment:
   fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] xintongsong closed pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
xintongsong closed pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask
URL: https://github.com/apache/flink/pull/21560


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1062396153


##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/CachedShuffleDescriptors.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.deployment;
+
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor.MaybeOffloaded;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptorFactory.ShuffleDescriptorAndIndex;
+import org.apache.flink.runtime.executiongraph.IntermediateResultPartition;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.shuffle.ShuffleDescriptor;
+import org.apache.flink.util.function.FunctionWithException;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** {@link ShuffleDescriptor}s cache for a {@link ConsumedPartitionGroup}. */
+public class CachedShuffleDescriptors {
+    /**
+     * Stores all serialized shuffle descriptors with indexes. For a partition, it may be added a
+     * serialized unknown shuffle descriptor to this list first, and then added the real descriptor
+     * later.
+     */
+    private final List<MaybeOffloaded<ShuffleDescriptorAndIndex[]>> serializedShuffleDescriptors;
+
+    /**
+     * Stores all to be serialized shuffle descriptors, They will be serialized and added to
+     * serializedShuffleDescriptors during the next time TaskDeploymentDescriptor is generated.
+     */
+    private final Queue<ShuffleDescriptorAndIndex> toBeSerialized;
+
+    /** Stores the mapping of resultPartitionId to index subscripts in consumed partition group. */
+    private final Map<IntermediateResultPartitionID, Integer> resultPartitionIdToIndex;
+
+    public CachedShuffleDescriptors(
+            ConsumedPartitionGroup consumedPartitionGroup,
+            MaybeOffloaded<ShuffleDescriptorAndIndex[]> initialSerializedShuffleDescriptors) {
+        this.resultPartitionIdToIndex = new HashMap<>();
+        int index = 0;
+        for (IntermediateResultPartitionID resultPartitionID : consumedPartitionGroup) {
+            resultPartitionIdToIndex.put(resultPartitionID, index++);
+        }
+        this.toBeSerialized = new ArrayDeque<>(consumedPartitionGroup.size());
+        this.serializedShuffleDescriptors = new ArrayList<>();
+        this.serializedShuffleDescriptors.add(initialSerializedShuffleDescriptors);
+    }
+
+    public MaybeOffloaded<ShuffleDescriptorAndIndex[]> getShuffleDescriptors(int index) {

Review Comment:
   Using `getAllSerializedShuffleDescriptors` to replace these two method's invoke and remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1062442217


##########
flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java:
##########
@@ -650,24 +650,46 @@ public enum SchedulerType {
                     .withDescription(
                             "The JobManager's ResourceID. If not configured, the ResourceID will be generated randomly.");
 
+    /** Type of consume partition mode of hybrid edge. */
+    public enum HybridConsumePartitionMode {
+        ONLY_CONSUME_ALL_FINISHED(true),
+        CAN_CONSUME_PARTIAL_FINISHED(true),
+        CAN_CONSUME_UN_FINISHED(false);

Review Comment:
   changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1060305155


##########
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/InternalExecutionGraphAccessor.java:
##########
@@ -122,4 +122,6 @@ List<ShuffleDescriptor> getClusterPartitionShuffleDescriptors(
             IntermediateDataSetID intermediateResultPartition);
 
     MarkPartitionFinishedStrategy getMarkPartitionFinishedStrategy();
+
+    boolean isHybridEnableConsumePartialFinishedProducer();

Review Comment:
   Yes, I also didn't find a better way to handle this. Let's keep it first, If there is a better way, we can reconstruct it later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] xintongsong commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
xintongsong commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1062282342


##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/TaskDeploymentDescriptorFactory.java:
##########
@@ -412,4 +430,28 @@ public static PartitionLocationConstraint fromJobType(JobType jobType) {
             }
         }
     }
+
+    /**
+     * This class represents the shuffle descriptor with it index in {@link ConsumedPartitionGroup}.
+     */
+    public static class ShuffleDescriptorAndIndex implements Serializable {
+        private static final long serialVersionUID = 852181945034989215L;

Review Comment:
   Serial version uid should start with `1L`.



##########
flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java:
##########
@@ -650,24 +650,46 @@ public enum SchedulerType {
                     .withDescription(
                             "The JobManager's ResourceID. If not configured, the ResourceID will be generated randomly.");
 
+    /** Type of consume partition mode of hybrid edge. */
+    public enum HybridConsumePartitionMode {
+        ONLY_CONSUME_ALL_FINISHED(true),
+        CAN_CONSUME_PARTIAL_FINISHED(true),
+        CAN_CONSUME_UN_FINISHED(false);
+
+        private final boolean onlyConsumeFinishedPartition;
+
+        HybridConsumePartitionMode(boolean onlyConsumeFinishedPartition) {
+            this.onlyConsumeFinishedPartition = onlyConsumeFinishedPartition;
+        }
+
+        public boolean isOnlyConsumeFinishedPartition() {
+            return onlyConsumeFinishedPartition;
+        }
+    }
+
     @Documentation.Section({
         Documentation.Sections.EXPERT_SCHEDULING,
         Documentation.Sections.ALL_JOB_MANAGER
     })
-    public static final ConfigOption<Boolean> ONLY_CONSUME_FINISHED_PARTITION =
-            key("jobmanager.partition.hybrid.only-consume-finished-partition")
-                    .booleanType()
+    public static final ConfigOption<HybridConsumePartitionMode> HYBRID_CONSUME_PARTITION_MODE =
+            key("jobmanager.partition.hybrid.consume-partition-mode")
+                    .enumType(HybridConsumePartitionMode.class)
                     .noDefaultValue()
                     .withDescription(
                             Description.builder()
                                     .text(
-                                            "Controls whether the scheduler only allows downstream task consume finished partition. "
-                                                    + "Note that this option is allowed only when %s has been set to %s, "
-                                                    + "and if you also enable speculative execution(%s has been set to true),"
-                                                    + "this option can only be set to true.",
+                                            "Controls how the downstream task consumes hybrid type producer edge. "
+                                                    + "Note that this option is allowed only when %s has been set to %s. "
+                                                    + "Accepted values are:",
                                             code(SCHEDULER.key()),
-                                            code(SchedulerType.AdaptiveBatch.name()),
-                                            code(SPECULATIVE_ENABLED.key()))
+                                            code(SchedulerType.AdaptiveBatch.name()))
+                                    .list(
+                                            text(
+                                                    "'ONLY_CONSUME_ALL_FINISHED': Downstream tasks can consume data only when all producers are finished."),

Review Comment:
   Use `HybridConsumePartitionMode.CAN_CONSUME_PARTIAL_FINISHED.name()`.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/CachedShuffleDescriptors.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.deployment;
+
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor.MaybeOffloaded;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptorFactory.ShuffleDescriptorAndIndex;
+import org.apache.flink.runtime.executiongraph.IntermediateResultPartition;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.shuffle.ShuffleDescriptor;
+import org.apache.flink.util.function.FunctionWithException;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** {@link ShuffleDescriptor}s cache for a {@link ConsumedPartitionGroup}. */
+public class CachedShuffleDescriptors {
+    /**
+     * Stores all serialized shuffle descriptors with indexes. For a partition, it may be added a
+     * serialized unknown shuffle descriptor to this list first, and then added the real descriptor
+     * later.
+     */
+    private final List<MaybeOffloaded<ShuffleDescriptorAndIndex[]>> serializedShuffleDescriptors;
+
+    /**
+     * Stores all to be serialized shuffle descriptors, They will be serialized and added to
+     * serializedShuffleDescriptors during the next time TaskDeploymentDescriptor is generated.
+     */
+    private final Queue<ShuffleDescriptorAndIndex> toBeSerialized;
+
+    /** Stores the mapping of resultPartitionId to index subscripts in consumed partition group. */
+    private final Map<IntermediateResultPartitionID, Integer> resultPartitionIdToIndex;
+
+    public CachedShuffleDescriptors(
+            ConsumedPartitionGroup consumedPartitionGroup,
+            MaybeOffloaded<ShuffleDescriptorAndIndex[]> initialSerializedShuffleDescriptors) {
+        this.resultPartitionIdToIndex = new HashMap<>();
+        int index = 0;
+        for (IntermediateResultPartitionID resultPartitionID : consumedPartitionGroup) {
+            resultPartitionIdToIndex.put(resultPartitionID, index++);
+        }
+        this.toBeSerialized = new ArrayDeque<>(consumedPartitionGroup.size());
+        this.serializedShuffleDescriptors = new ArrayList<>();
+        this.serializedShuffleDescriptors.add(initialSerializedShuffleDescriptors);
+    }
+
+    public MaybeOffloaded<ShuffleDescriptorAndIndex[]> getShuffleDescriptors(int index) {

Review Comment:
   The concept `index` is confusing, because the same term has different semantics in `resultPartitionIdToIndex` and here.



##########
flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java:
##########
@@ -650,24 +650,46 @@ public enum SchedulerType {
                     .withDescription(
                             "The JobManager's ResourceID. If not configured, the ResourceID will be generated randomly.");
 
+    /** Type of consume partition mode of hybrid edge. */
+    public enum HybridConsumePartitionMode {
+        ONLY_CONSUME_ALL_FINISHED(true),
+        CAN_CONSUME_PARTIAL_FINISHED(true),
+        CAN_CONSUME_UN_FINISHED(false);

Review Comment:
   ```
   public enum HybridPartitionDataConsumeConstraint {
       ALL_PRODUCERS_FINISHED,
       ONLY_FINISHED_PRODUCERS,
       UNFINISHED_PRODUCERS
   }
   ```



##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/CachedShuffleDescriptors.java:
##########
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.deployment;
+
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor.MaybeOffloaded;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptorFactory.ShuffleDescriptorAndIndex;
+import org.apache.flink.runtime.executiongraph.IntermediateResultPartition;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.shuffle.ShuffleDescriptor;
+import org.apache.flink.util.function.FunctionWithException;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** {@link ShuffleDescriptor}s cache for a {@link ConsumedPartitionGroup}. */
+public class CachedShuffleDescriptors {
+    /**
+     * Stores all serialized shuffle descriptors with indexes. For a partition, it may be added a
+     * serialized unknown shuffle descriptor to this list first, and then added the real descriptor
+     * later.
+     */
+    private final List<MaybeOffloaded<ShuffleDescriptorAndIndex[]>> serializedShuffleDescriptors;
+
+    /**
+     * Stores all to be serialized shuffle descriptors, They will be serialized and added to
+     * serializedShuffleDescriptors during the next time TaskDeploymentDescriptor is generated.
+     */
+    private final Queue<ShuffleDescriptorAndIndex> toBeSerialized;
+
+    /** Stores the mapping of resultPartitionId to index subscripts in consumed partition group. */
+    private final Map<IntermediateResultPartitionID, Integer> resultPartitionIdToIndex;
+
+    public CachedShuffleDescriptors(
+            ConsumedPartitionGroup consumedPartitionGroup,
+            MaybeOffloaded<ShuffleDescriptorAndIndex[]> initialSerializedShuffleDescriptors) {

Review Comment:
   I wonder if we can pass in unserialized shuffle descriptors and serialize them inside the cache. Currently, we have the descriptors serialized both outside and inside the cache, which is a bit duplicated.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/CachedShuffleDescriptors.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.deployment;
+
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor.MaybeOffloaded;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptorFactory.ShuffleDescriptorAndIndex;
+import org.apache.flink.runtime.executiongraph.IntermediateResultPartition;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.shuffle.ShuffleDescriptor;
+import org.apache.flink.util.function.FunctionWithException;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** {@link ShuffleDescriptor}s cache for a {@link ConsumedPartitionGroup}. */
+public class CachedShuffleDescriptors {
+    /**
+     * Stores all serialized shuffle descriptors with indexes. For a partition, it may be added a
+     * serialized unknown shuffle descriptor to this list first, and then added the real descriptor
+     * later.
+     */
+    private final List<MaybeOffloaded<ShuffleDescriptorAndIndex[]>> serializedShuffleDescriptors;
+
+    /**
+     * Stores all to be serialized shuffle descriptors, They will be serialized and added to
+     * serializedShuffleDescriptors during the next time TaskDeploymentDescriptor is generated.
+     */
+    private final Queue<ShuffleDescriptorAndIndex> toBeSerialized;
+
+    /** Stores the mapping of resultPartitionId to index subscripts in consumed partition group. */
+    private final Map<IntermediateResultPartitionID, Integer> resultPartitionIdToIndex;
+
+    public CachedShuffleDescriptors(
+            ConsumedPartitionGroup consumedPartitionGroup,
+            MaybeOffloaded<ShuffleDescriptorAndIndex[]> initialSerializedShuffleDescriptors) {
+        this.resultPartitionIdToIndex = new HashMap<>();
+        int index = 0;
+        for (IntermediateResultPartitionID resultPartitionID : consumedPartitionGroup) {
+            resultPartitionIdToIndex.put(resultPartitionID, index++);
+        }
+        this.toBeSerialized = new ArrayDeque<>(consumedPartitionGroup.size());
+        this.serializedShuffleDescriptors = new ArrayList<>();
+        this.serializedShuffleDescriptors.add(initialSerializedShuffleDescriptors);
+    }
+
+    public MaybeOffloaded<ShuffleDescriptorAndIndex[]> getShuffleDescriptors(int index) {

Review Comment:
   I think we don't need `getShuffleDescriptors(int index)` and `getSerializedShuffleDescriptorsSize()` at all. They are only used in a for-loop in `IntermediateResult`, where we can simply use `getAllSerializedShuffleDescriptors()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #21560: [FLINK-29768] Hybrid shuffle supports consume partial finished subtask

Posted by GitBox <gi...@apache.org>.
reswqa commented on code in PR #21560:
URL: https://github.com/apache/flink/pull/21560#discussion_r1062395194


##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/TaskDeploymentDescriptorFactory.java:
##########
@@ -412,4 +430,28 @@ public static PartitionLocationConstraint fromJobType(JobType jobType) {
             }
         }
     }
+
+    /**
+     * This class represents the shuffle descriptor with it index in {@link ConsumedPartitionGroup}.
+     */
+    public static class ShuffleDescriptorAndIndex implements Serializable {
+        private static final long serialVersionUID = 852181945034989215L;

Review Comment:
   fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org