You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "RanJinh (via GitHub)" <gi...@apache.org> on 2023/07/15 09:38:28 UTC

[GitHub] [flink] RanJinh opened a new pull request, #23000: [runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

RanJinh opened a new pull request, #23000:
URL: https://github.com/apache/flink/pull/23000

   ## What is the purpose of the change
   
    This pull request allows using blocking ResultPartitionType in streaming execution mode if operator only outputs records on end of input.
   
   
   ## Brief change log
   
     - Add an internal method to `StreamOperator`  to allow implementations to report whether the operator will output records on end of input.
   - Add an internal method to `Function`  to define whether the user-defined function will output records on end of input.
   - Add an `EOFCoGroupOperator` to optimize the DataStream#coGroup when output on end of input.
   - Optimize job to schedule and execute the output-on-EOF operator as well as its upstream operators in batch mode.
   
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
     - Add integration tests `OperatorOutputEOFITCase` and unit tests in `StreamingJobGraphGeneratorTest` to confirm that, in `RuntimeExecutionMode.STREAMING`, the operator has been schedule correctly and the downstream node would not be set up before the output-on-EOF operator node has finished.
     - Add the unit tests in `StreamGraphGeneratorTest` and `StreamGraphGeneratorBatchExecutionTest`  to confirm that the operator  will set `InputRequirement` correctly.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature?  yes
     - If yes, how is the feature documented?  not documented


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] flinkbot commented on pull request #23000: [runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #23000:
URL: https://github.com/apache/flink/pull/23000#issuecomment-1636722337

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "56609726e9384710c192eeaa851525c574059515",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "56609726e9384710c192eeaa851525c574059515",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 56609726e9384710c192eeaa851525c574059515 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] WencongLiu commented on a diff in pull request #23000: [FLINK-32594][runtime] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

Posted by "WencongLiu (via GitHub)" <gi...@apache.org>.
WencongLiu commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1321166055


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SimpleOperatorFactory.java:
##########
@@ -132,4 +134,12 @@ public void setInputType(TypeInformation<?> type, ExecutionConfig executionConfi
     public Class<? extends StreamOperator> getStreamOperatorClass(ClassLoader classLoader) {
         return operator.getClass();
     }
+
+    @Override
+    public OperatorAttributes getOperatorAttributes() {
+        if (operatorAttributes == null) {
+            operatorAttributes = operator.getOperatorAttributes();
+        }
+        return operatorAttributes;

Review Comment:
   If the `OperatorAttributes ` is a concept inside `StreamOperator`, why we put the `operatorAttributes` in the `SimpleOperatorFactory` ? Currently it's always unused in `SimpleOperatorFactory`.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorAttributes.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.annotation.Experimental;
+
+/**
+ * OperatorAttributes element provides Job Manager with information that can be used to optimize job
+ * performance.
+ */
+@Experimental
+public class OperatorAttributes {
+
+    private final boolean outputOnEOF;
+    private final boolean internalSorterSupported;
+
+    public OperatorAttributes(boolean isOutputOnEOF, boolean internalSorterSupported) {
+        this.outputOnEOF = isOutputOnEOF;
+        this.internalSorterSupported = internalSorterSupported;
+    }
+
+    /**
+     * Returns true iff the operator can only emit records after inputs have reached EOF.
+     *
+     * <p>Here are the implications when it is true:
+     *
+     * <ul>
+     *   <li>The results of this operator as well as its upstream operators have blocking partition
+     *       type.
+     *   <li>This operator as well as its upstream operators will be executed in batch mode.
+     * </ul>
+     */
+    public boolean isOutputOnEOF() {
+        return outputOnEOF;
+    }
+
+    /**
+     * Returns true iff the operator uses an internal sorter to sort inputs by key when any of the

Review Comment:
   `iff` -> `if`



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorAttributesBuilder.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+/** The builder class for {@link OperatorAttributes}. */
+@Experimental
+public class OperatorAttributesBuilder {
+
+    private static final Logger LOG = LoggerFactory.getLogger(StreamingJobGraphGenerator.class);

Review Comment:
   `private static final Logger LOG = LoggerFactory.getLogger(OperatorAttributesBuilder.class);`



##########
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/datastream/CoGroupedStreamsTest.java:
##########
@@ -81,4 +82,20 @@ public void testSetAllowedLateness() {
         Assert.assertEquals(
                 lateness.toMilliseconds(), withLateness.getAllowedLateness().toMilliseconds());
     }
+
+    @Test
+    public void testEndOfStreamWindowsCoGroup() {

Review Comment:
   if there should be another test `testEndOfStreamWindowsAggregate`?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2035,7 +2039,11 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            scheduleTriggerWithDelay(clock.relativeTimeMillis(), getRandomInitDelay());
+            if (withRandomDelay) {
+                scheduleTriggerWithDelay(clock.relativeTimeMillis(), getRandomInitDelay());
+            } else {

Review Comment:
   Could you please explain why this change is needed? 🤔 Thanks.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/EOFAggregationOperator.java:
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.sort;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.AlgorithmOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.core.memory.ManagedMemoryUseCase;
+import org.apache.flink.runtime.io.disk.iomanager.IOManager;
+import org.apache.flink.runtime.memory.MemoryAllocationException;
+import org.apache.flink.runtime.memory.MemoryManager;
+import org.apache.flink.runtime.operators.sort.ExternalSorter;
+import org.apache.flink.runtime.operators.sort.PushSorter;
+import org.apache.flink.runtime.util.NonReusingKeyGroupedIterator;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.MutableObjectIterator;
+
+/**
+ * This operator would replace the regular Aggregate process when the WindowAssigner is
+ * EndOfStreamWindows.
+ */
+@Internal
+public class EOFAggregationOperator<IN, KEY, ACC, OUT>
+        extends AbstractUdfStreamOperator<OUT, AggregateFunction<IN, ACC, OUT>>
+        implements OneInputStreamOperator<IN, OUT>, BoundedOneInput {
+
+    private PushSorter<Tuple2<byte[], StreamRecord<IN>>> sorter;
+    TypeSerializer<Tuple2<byte[], StreamRecord<IN>>> keyAndValueSerializer;
+    TypeComparator<Tuple2<byte[], StreamRecord<IN>>> comparator;
+    private KeySelector<IN, KEY> keySelector;
+    private TypeSerializer<KEY> keySerializer;
+    private DataOutputSerializer dataOutputSerializer;
+    private long lastWatermarkTimestamp = Long.MIN_VALUE;
+
+    public EOFAggregationOperator(AggregateFunction<IN, ACC, OUT> userFunction) {
+        super(userFunction);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<OUT>> output) {
+        super.setup(containingTask, config, output);
+        ClassLoader userCodeClassLoader = containingTask.getUserCodeClassLoader();
+        MemoryManager memoryManager = containingTask.getEnvironment().getMemoryManager();
+        IOManager ioManager = containingTask.getEnvironment().getIOManager();
+
+        keySelector = (KeySelector<IN, KEY>) config.getStatePartitioner(0, userCodeClassLoader);
+        keySerializer = config.getStateKeySerializer(userCodeClassLoader);
+        int keyLength = keySerializer.getLength();
+
+        TypeSerializer<IN> typeSerializerA = config.getTypeSerializerIn(0, userCodeClassLoader);
+        keyAndValueSerializer = new KeyAndValueSerializer<>(typeSerializerA, keyLength);
+
+        if (keyLength > 0) {
+            dataOutputSerializer = new DataOutputSerializer(keyLength);
+            comparator = new FixedLengthByteKeyComparator<>(keyLength);
+        } else {
+            dataOutputSerializer = new DataOutputSerializer(64);
+            comparator = new VariableLengthByteKeyComparator<>();
+        }
+
+        ExecutionConfig executionConfig = containingTask.getEnvironment().getExecutionConfig();
+
+        double managedMemoryFraction =
+                config.getManagedMemoryFractionOperatorUseCaseOfSlot(
+                        ManagedMemoryUseCase.OPERATOR,
+                        containingTask.getEnvironment().getTaskConfiguration(),
+                        userCodeClassLoader);
+
+        Configuration jobConfiguration = containingTask.getEnvironment().getJobConfiguration();
+
+        try {
+            sorter =
+                    ExternalSorter.newBuilder(
+                                    memoryManager,
+                                    containingTask,
+                                    keyAndValueSerializer,
+                                    comparator,
+                                    executionConfig)
+                            .memoryFraction(managedMemoryFraction)
+                            .enableSpilling(
+                                    ioManager,
+                                    jobConfiguration.get(AlgorithmOptions.SORT_SPILLING_THRESHOLD))
+                            .maxNumFileHandles(
+                                    jobConfiguration.get(AlgorithmOptions.SPILLING_MAX_FAN))
+                            .objectReuse(executionConfig.isObjectReuseEnabled())
+                            .largeRecords(
+                                    jobConfiguration.get(
+                                            AlgorithmOptions.USE_LARGE_RECORDS_HANDLER))
+                            .build();
+        } catch (MemoryAllocationException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    @Override
+    public void endInput() throws Exception {
+        sorter.finishReading();
+        MutableObjectIterator<Tuple2<byte[], StreamRecord<IN>>> iterator = sorter.getIterator();
+        TimestampedCollector<OUT> outputCollector = new TimestampedCollector<>(output);
+        NonReusingKeyGroupedIterator<Tuple2<byte[], StreamRecord<IN>>> keyGroupedIterator =
+                new NonReusingKeyGroupedIterator<>(iterator, comparator);
+
+        while (keyGroupedIterator.nextKey()) {
+            ACC acc = userFunction.createAccumulator();
+            NonReusingKeyGroupedIterator<Tuple2<byte[], StreamRecord<IN>>>.ValuesIterator records =
+                    keyGroupedIterator.getValues();
+            for (Tuple2<byte[], StreamRecord<IN>> record : records) {
+                acc = userFunction.add(record.f1.getValue(), acc);
+            }
+            outputCollector.collect(userFunction.getResult(acc));
+        }
+
+        Watermark watermark = new Watermark(lastWatermarkTimestamp);
+        if (getTimeServiceManager().isPresent()) {
+            getTimeServiceManager().get().advanceWatermark(watermark);

Review Comment:
   If the user choose to use `EndOfStreamWindows`, there shouldn't be any registered timers. As a result, the TimeService may be useless. WDYT?



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorBuilder.java:
##########
@@ -346,4 +347,10 @@ public String generateOperatorDescription(Function function1, @Nullable Function
     public long getAllowedLateness() {
         return allowedLateness;
     }
+
+    public boolean canApplyEndOfStreamOperator() {
+        return windowAssigner instanceof EndOfStreamWindows
+                && trigger instanceof EndOfStreamWindows.EndOfStreamTrigger
+                && evictor == null;

Review Comment:
   I think we'd better check that we shouldn't use user-defined trigger if the `windowAssigner `  is `EndOfStreamWindows` in `public void trigger(Trigger<? super T, ? super W> trigger) {`
   
   Currently, `EndOfStreamWindows` and `EndOfStreamTrigger` are a fixed combination, right?



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorAttributes.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.annotation.Experimental;
+
+/**
+ * OperatorAttributes element provides Job Manager with information that can be used to optimize job
+ * performance.

Review Comment:
   It is preferable to specify the specific aspect or dimension of the job that can be optimized.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorAttributes.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.annotation.Experimental;
+
+/**
+ * OperatorAttributes element provides Job Manager with information that can be used to optimize job
+ * performance.
+ */
+@Experimental
+public class OperatorAttributes {
+
+    private final boolean outputOnEOF;
+    private final boolean internalSorterSupported;
+
+    public OperatorAttributes(boolean isOutputOnEOF, boolean internalSorterSupported) {
+        this.outputOnEOF = isOutputOnEOF;
+        this.internalSorterSupported = internalSorterSupported;
+    }
+
+    /**
+     * Returns true iff the operator can only emit records after inputs have reached EOF.

Review Comment:
   `iff` -> `if`



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:
##########
@@ -1472,15 +1500,35 @@ private ResultPartitionType getResultPartitionType(StreamEdge edge) {
             case HYBRID_SELECTIVE:
                 return ResultPartitionType.HYBRID_SELECTIVE;
             case UNDEFINED:
-                return determineUndefinedResultPartitionType(edge.getPartitioner());
+                return determineUndefinedResultPartitionType(edge);
             default:
                 throw new UnsupportedOperationException(
                         "Data exchange mode " + edge.getExchangeMode() + " is not supported yet.");
         }
     }
 
-    private ResultPartitionType determineUndefinedResultPartitionType(
-            StreamPartitioner<?> partitioner) {
+    private ResultPartitionType determineUndefinedResultPartitionType(StreamEdge edge) {
+        if (outputBlockingNodesID.contains(edge.getSourceId())) {
+            // use corresponding partition type if the upstream node outputs EOF
+            edge.setBufferTimeout(ExecutionOptions.DISABLED_NETWORK_BUFFER_TIMEOUT);
+            if (!outputBlockingNodesID.contains(edge.getTargetId())) {
+                return ResultPartitionType.BLOCKING;
+            }

Review Comment:
   `            if (!outputBlockingNodesID.contains(edge.getTargetId())) {
                   return ResultPartitionType.BLOCKING;
               }`
     Why we need to check the downstream operator when the outputBlockingNodesID contains `edge.getSourceId()` ?



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/OneInputTransformationTranslator.java:
##########
@@ -50,24 +50,47 @@ public Collection<Integer> translateForBatchInternal(
                         keySelector,
                         transformation.getStateKeyType(),
                         context);
-        boolean isKeyed = keySelector != null;
-        if (isKeyed) {
-            BatchExecutionUtils.applyBatchExecutionSettings(
-                    transformation.getId(), context, StreamConfig.InputRequirement.SORTED);
-        }
+
+        maybeApplyBatchExecutionSettings(transformation, context);
 
         return ids;
     }
 
     @Override
     public Collection<Integer> translateForStreamingInternal(
             final OneInputTransformation<IN, OUT> transformation, final Context context) {
-        return translateInternal(
-                transformation,
-                transformation.getOperatorFactory(),
-                transformation.getInputType(),
-                transformation.getStateKeySelector(),
-                transformation.getStateKeyType(),
-                context);
+        Collection<Integer> ids =
+                translateInternal(
+                        transformation,
+                        transformation.getOperatorFactory(),
+                        transformation.getInputType(),
+                        transformation.getStateKeySelector(),
+                        transformation.getStateKeyType(),
+                        context);
+
+        if (transformation.isOutputOnEOF()) {
+            // Try to apply batch execution settings for streaming mode transformation.
+            maybeApplyBatchExecutionSettings(transformation, context);
+        }
+
+        return ids;
+    }
+
+    private void maybeApplyBatchExecutionSettings(
+            final OneInputTransformation<IN, OUT> transformation, final Context context) {
+        KeySelector<IN, ?> keySelector = transformation.getStateKeySelector();
+        boolean isKeyed = keySelector != null;
+        if (isKeyed) {
+            StreamConfig.InputRequirement inputRequirement;
+            if (!transformation.isInternalSorterSupported()) {
+                inputRequirement = StreamConfig.InputRequirement.SORTED;
+            } else {
+                // The inputs of transformation don't require InputRequirement.SORTED
+                // if the transformation support internal sorter.
+                inputRequirement = StreamConfig.InputRequirement.PASS_THROUGH;
+            }

Review Comment:
   Actually, these logic above can also covered by `BatchExecutionUtils#getInputRequirements` like `TwoInputTransformationTranslator` and `MultiInputTransformationTranslator`.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2035,7 +2039,11 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            scheduleTriggerWithDelay(clock.relativeTimeMillis(), getRandomInitDelay());
+            if (withRandomDelay) {
+                scheduleTriggerWithDelay(clock.relativeTimeMillis(), getRandomInitDelay());
+            } else {

Review Comment:
   Could you please explain why this change is needed? 🤔 Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-32594][runtime]Support EndOfStreamTrigger and isOutputOnEOF operator attribute to optimize task deployment [flink]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23000:
URL: https://github.com/apache/flink/pull/23000#issuecomment-2595362747

   This PR is being marked as stale since it has not had any activity in the last 180 days. 
   If you would like to keep this PR alive, please leave a comment asking for a review. 
   If the PR has merge conflicts, update it with the latest from the base branch.
   
   If you are having difficulty finding a reviewer, please reach out to the 
   community, contact details can be found here: https://flink.apache.org/what-is-flink/community/
   
   If this PR is no longer valid or desired, please feel free to close it. 
   If no activity occurs in the next 90 days, it will be automatically closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on PR #23000:
URL: https://github.com/apache/flink/pull/23000#issuecomment-1647405728

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-32594][runtime]Support EndOfStreamTrigger and isOutputOnEOF operator attribute to optimize task deployment [flink]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23000:
URL: https://github.com/apache/flink/pull/23000#issuecomment-2774620553

   This PR has been closed since it has not had any activity in 120 days. 
   If you feel like this was a mistake, or you would like to continue working on it, 
   please feel free to re-open the PR and ask for a review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-32594][runtime]Support EndOfStreamTrigger and isOutputOnEOF operator attribute to optimize task deployment [flink]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #23000: [FLINK-32594][runtime]Support EndOfStreamTrigger and isOutputOnEOF operator attribute to optimize task deployment
URL: https://github.com/apache/flink/pull/23000


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1307025612


##########
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkTransformationTranslatorITCase.java:
##########
@@ -225,7 +225,8 @@ private StreamNode findWriter(StreamGraph streamGraph) {
     }
 
     private StreamNode findCommitter(StreamGraph streamGraph) {
-        return findNodeName(streamGraph, name -> name.contains("Committer"));
+        return findNodeName(

Review Comment:
   > Can you explain why we need this change?
   
   This test is unstable when ITCase runs in IDEA that `Global Committer` node could be wrongly found as `Committer`. This change may solve it to avoid risk. Do you think it is necessary?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] WencongLiu commented on a diff in pull request #23000: [FLINK-32594][runtime] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

Posted by "WencongLiu (via GitHub)" <gi...@apache.org>.
WencongLiu commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1321042128


##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java:
##########
@@ -1014,7 +1014,7 @@ public void resumeConsumption(InputChannelInfo channelInfo) throws IOException {
     @Override
     public void acknowledgeAllRecordsProcessed(InputChannelInfo channelInfo) throws IOException {
         checkState(!isFinished(), "InputGate already finished.");
-        if (!enabledTieredStorage()) {
+        if (consumedPartitionType.isPipelinedOrPipelinedBoundedResultPartition()) {

Review Comment:
   Why the `SingleInputGate` only needs to acknowledge that all records are processed when the partition type is pipelined?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on PR #23000:
URL: https://github.com/apache/flink/pull/23000#issuecomment-1650824543

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] WencongLiu commented on a diff in pull request #23000: [FLINK-32594][runtime] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

Posted by "WencongLiu (via GitHub)" <gi...@apache.org>.
WencongLiu commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1321179253


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorAttributes.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.annotation.Experimental;
+
+/**
+ * OperatorAttributes element provides Job Manager with information that can be used to optimize job
+ * performance.
+ */
+@Experimental
+public class OperatorAttributes {
+
+    private final boolean outputOnEOF;
+    private final boolean internalSorterSupported;
+
+    public OperatorAttributes(boolean isOutputOnEOF, boolean internalSorterSupported) {
+        this.outputOnEOF = isOutputOnEOF;
+        this.internalSorterSupported = internalSorterSupported;
+    }
+
+    /**
+     * Returns true iff the operator can only emit records after inputs have reached EOF.

Review Comment:
   `iff` -> `if`



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorAttributes.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.annotation.Experimental;
+
+/**
+ * OperatorAttributes element provides Job Manager with information that can be used to optimize job
+ * performance.
+ */
+@Experimental
+public class OperatorAttributes {
+
+    private final boolean outputOnEOF;
+    private final boolean internalSorterSupported;
+
+    public OperatorAttributes(boolean isOutputOnEOF, boolean internalSorterSupported) {
+        this.outputOnEOF = isOutputOnEOF;
+        this.internalSorterSupported = internalSorterSupported;
+    }
+
+    /**
+     * Returns true iff the operator can only emit records after inputs have reached EOF.
+     *
+     * <p>Here are the implications when it is true:
+     *
+     * <ul>
+     *   <li>The results of this operator as well as its upstream operators have blocking partition
+     *       type.
+     *   <li>This operator as well as its upstream operators will be executed in batch mode.
+     * </ul>
+     */
+    public boolean isOutputOnEOF() {
+        return outputOnEOF;
+    }
+
+    /**
+     * Returns true iff the operator uses an internal sorter to sort inputs by key when any of the

Review Comment:
   `iff` -> `if`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1267888031


##########
flink-core/src/main/java/org/apache/flink/api/dag/Transformation.java:
##########
@@ -637,4 +637,16 @@ public int hashCode() {
         result = 31 * result + (int) (bufferTimeout ^ (bufferTimeout >>> 32));
         return result;
     }
+

Review Comment:
   Yeah, you are right. I will modify this, and we should also update the FLIP.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1268184648


##########
flink-core/src/main/java/org/apache/flink/api/common/functions/Function.java:
##########
@@ -27,4 +28,16 @@
  * method) interfaces that can be implemented via Java 8 lambdas.
  */
 @Public
-public interface Function extends java.io.Serializable {}
+public interface Function extends java.io.Serializable {

Review Comment:
   Related change is in this reply: https://github.com/apache/flink/pull/23000#discussion_r1267862410



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Sxnan commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "Sxnan (via GitHub)" <gi...@apache.org>.
Sxnan commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1266320558


##########
flink-core/src/main/java/org/apache/flink/api/common/functions/Function.java:
##########
@@ -27,4 +28,16 @@
  * method) interfaces that can be implemented via Java 8 lambdas.
  */
 @Public
-public interface Function extends java.io.Serializable {}
+public interface Function extends java.io.Serializable {

Review Comment:
   I think we don't need to have the `isOutputEOF` and `internalSOrterSupported` for the `Function` interface, as the function is not able to emit records after the inputs reach EOF or internally sort the inputs by key, which means that these two methods always return false.
   
   The `isOutputEOF` and `internalSorterSupported` is the concept of an operator and can be determined by the `AbstractUdfStreamOperator` itself.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] reswqa commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1267509042


##########
flink-core/src/main/java/org/apache/flink/api/common/functions/Function.java:
##########
@@ -27,4 +28,16 @@
  * method) interfaces that can be implemented via Java 8 lambdas.
  */
 @Public
-public interface Function extends java.io.Serializable {}
+public interface Function extends java.io.Serializable {
+    /** Returns true iff the operator can only emit records after inputs have reached EOF. */
+    @Internal
+    default boolean isOutputEOF() {

Review Comment:
   Why a common `Function` touch the concepts related to `operator`? This makes me feel confused. 🤔 



##########
flink-core/src/main/java/org/apache/flink/api/common/functions/Function.java:
##########
@@ -27,4 +28,16 @@
  * method) interfaces that can be implemented via Java 8 lambdas.
  */
 @Public
-public interface Function extends java.io.Serializable {}
+public interface Function extends java.io.Serializable {
+    /** Returns true iff the operator can only emit records after inputs have reached EOF. */
+    @Internal
+    default boolean isOutputEOF() {

Review Comment:
   Why a common `Function` touch the concepts related to `operator`? This makes me feel confused. 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on PR #23000:
URL: https://github.com/apache/flink/pull/23000#issuecomment-1637435604

   I added the `org.apache.flink.streaming.api.windowing.assigners.EndOfStreamWindows.assignWindows` in the `archunit-violations` file to avid `org.apache.flink.architecture.ArchitectureTest` detecting the annotation, which also be done for other window assigners.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1300982941


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -581,6 +581,14 @@ private void startTriggeringCheckpoint(CheckpointTriggerRequest request) {
 
             CompletableFuture<Void> masterTriggerCompletionPromise = new CompletableFuture<>();
 
+            if (checkpointPlanFuture.get() == null) {

Review Comment:
   It seems unnecessary to keep calling `startTriggeringCheckpoint` when there is any task writing data to a blocking-typed partition.
   
   And it might be useful to immediately trigger a checkpoint right after it is allowed (e.g. no task is writing data to a blocking-typed partition) AND there is been more than `execution.checkpointing.interval` since the job is started.
   
   Maybe the JM should schedule checkpoint coordinator as appropriate based on the task status changes?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCheckpointPlanCalculator.java:
##########
@@ -156,6 +156,13 @@ private void checkTasksStarted(List<Execution> toTrigger) throws CheckpointExcep
      * @return The plan of this checkpoint.
      */
     private CheckpointPlan calculateWithAllTasksRunning() {
+        // Do not trigger checkpoint before all BATCH tasks have finished.
+        for (ExecutionVertex task : allTasks) {
+            if (!task.allOutputsPipelinedOrPipelinedBounded()) {

Review Comment:
   In addition to ensuring that there is no task producing to block-typed partition, do we also need to ensure that there is no task consuming from block-typed partition?
   
   Maybe we can let tasks report this `end-of-data-from-blocking-inputs` similar to how tasks report `end-of-data`. You can see `TaskExecutor#notifyEndOfData` for more information.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/WindowedStream.java:
##########
@@ -328,6 +329,12 @@ public <ACC, R> SingleOutputStreamOperator<R> aggregate(
                     "This aggregation function cannot be a RichFunction.");
         }
 
+        if (builder.isEndOfStreamWindow()) {

Review Comment:
   Instead of adding this extra API to read an attribute of `WindowOperatorBuilder`, would it be simpler to let `builder.aggregate()` return the `EOFAggregationOperator`?



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamNode.java:
##########
@@ -442,4 +442,11 @@ public void setSupportsConcurrentExecutionAttempts(
             boolean supportsConcurrentExecutionAttempts) {
         this.supportsConcurrentExecutionAttempts = supportsConcurrentExecutionAttempts;
     }
+
+    public boolean isOutputOnEOF() {
+        if (operatorFactory == null) {

Review Comment:
   `operatorFactory` is not marked `@Nullable` in `StreamNode`. I suppose it should be impossible to have a StreamNode with `operatorFactory == null`.
   
   Can you explain when we need this check?



##########
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamGraphGeneratorBatchExecutionTest.java:
##########
@@ -491,6 +494,55 @@ public void testCoFeedbackThrowsExceptionInBatch() {
         testNoSupportForIterationsInBatchHelper(coFeedbackTransformation);
     }
 
+    @Test
+    public void operatorSortWithoutRepetitionTest() {

Review Comment:
   Typically we expect the test method name to start with `test...` and shows on the key logic/API that this test covers.
   
   How about naming it something like `testInternalSorterAndOutputOnEOF`?



##########
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamGraphGeneratorTest.java:
##########
@@ -960,6 +962,51 @@ public void testCacheSideOutput() {
         verifyCacheConsumeNode(env, upstreamParallelism, cacheTransformation);
     }
 
+    @Test
+    public void testTransformOutputEOFOperator() {

Review Comment:
   It seems that this test aims to check "if an operator has isOutputOnEOF = true" and has keyed inputs, then its input should be sorted.
   
   It is a bit weird/redundant to have this test since there is no logical connection between input requirement and isOutputOnEOF's value.
   
   Note that it is in general impossible and unnecessary to test every combination of configurations. We typically only add tests whose input/output has logical connections.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/CoGroupedStreams.java:
##########
@@ -366,6 +368,19 @@ public <T> DataStream<T> apply(
             // clean the closure
             function = input1.getExecutionEnvironment().clean(function);
 
+            // Using EOFCoGroupOperator when the window assigner is EndOfStreamWindows
+            if (windowAssigner instanceof EndOfStreamWindows) {

Review Comment:
   Should we use `EOFCoGroupOperator` only when both `trigger` and `evictor` are null? Otherwise, we might produce wrong results when user provides trigger/evictor.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:
##########
@@ -1108,7 +1128,13 @@ private void setOperatorConfig(
 
         config.setTimeCharacteristic(streamGraph.getTimeCharacteristic());
 
-        final CheckpointConfig checkpointCfg = streamGraph.getCheckpointConfig();
+        final CheckpointConfig checkpointCfg;
+        if (outputBlockingNodesID.contains(vertexId)) {
+            // the output blocking vertex should disable checkpoint.
+            checkpointCfg = new CheckpointConfig();

Review Comment:
   It is not obvious that `new CheckpointConfig` means "checkpoint is disabled".
   
   It might be simpler and more readable to update `config.setCheckpointingEnabled(...)` in the following code.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:
##########
@@ -703,6 +716,13 @@ private List<StreamEdge> createChain(
                         chainEntryPoints);
             }
 
+            for (StreamEdge edge : currentNode.getOutEdges()) {
+                // transfer outputBlocking from all of downstream nodes
+                if (outputBlockingNodesID.contains(edge.getTargetId())) {
+                    outputBlockingNodesID.add(currentNodeId);

Review Comment:
   Suppose we have three nodes A, B, and C that are chained in order. Node C has `isOutputOnEOF = true`. The current PR ensures that nodes B and C will be added to `outputBlockingNodesID`. 
   
   Will node A also be added to `outputBlockingNodesID`?



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/WindowedStream.java:
##########
@@ -328,6 +329,12 @@ public <ACC, R> SingleOutputStreamOperator<R> aggregate(
                     "This aggregation function cannot be a RichFunction.");
         }
 
+        if (builder.isEndOfStreamWindow()) {
+            function = input.getExecutionEnvironment().clean(function);
+            EOFAggregationOperator<T, K, ACC, R> operator = new EOFAggregationOperator<>(function);
+            return input.transform("EOFAggregation", resultType, operator);

Review Comment:
   Instead of hardcoding `EOFAggregation` as the operator name, would it be simpler and more consistent with the existing code to use `builder.generateOperatorName()` as the operator name?
   
   Similarly, it seems more consistent to use `builder.generateOperatorDescription()` as the operator description.
   
   In general, unless there is clear benefit of changing behavior, it is useful to keep unchanged and re-use existing code as much as possible.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:
##########
@@ -703,6 +716,13 @@ private List<StreamEdge> createChain(
                         chainEntryPoints);
             }
 
+            for (StreamEdge edge : currentNode.getOutEdges()) {

Review Comment:
   Suppose we have three nodes A, B, and C that are chained in order. Node A has `isOutputOnEOF = true`. The current PR ensures that nodes A and B will be added to `outputBlockingNodesID`.
   
   Will node C also be added to `outputBlockingNodesID`?



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/EOFAggregationOperator.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.sort;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.AlgorithmOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.core.memory.ManagedMemoryUseCase;
+import org.apache.flink.runtime.io.disk.iomanager.IOManager;
+import org.apache.flink.runtime.memory.MemoryAllocationException;
+import org.apache.flink.runtime.memory.MemoryManager;
+import org.apache.flink.runtime.operators.sort.ExternalSorter;
+import org.apache.flink.runtime.operators.sort.PushSorter;
+import org.apache.flink.runtime.util.NonReusingKeyGroupedIterator;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.MutableObjectIterator;
+
+/**
+ * This operator would replace the regular Aggregate process when the WindowAssigner is
+ * EndOfStreamWindows.
+ */
+@Internal
+public class EOFAggregationOperator<IN, KEY, ACC, OUT>
+        extends AbstractUdfStreamOperator<OUT, AggregateFunction<IN, ACC, OUT>>
+        implements OneInputStreamOperator<IN, OUT>, BoundedMultiInput {
+
+    private PushSorter<Tuple2<byte[], StreamRecord<IN>>> sorter;
+    TypeSerializer<Tuple2<byte[], StreamRecord<IN>>> keyAndValueSerializer;
+    TypeComparator<Tuple2<byte[], StreamRecord<IN>>> comparator;
+    private KeySelector<IN, KEY> keySelector;
+    private TypeSerializer<KEY> keySerializer;
+    private DataOutputSerializer dataOutputSerializer;
+    private long lastWatermarkTimestamp = Long.MIN_VALUE;
+    private int remainingInputNum = 1;
+    private transient OperatorAttributes operatorAttributes;
+
+    public EOFAggregationOperator(AggregateFunction<IN, ACC, OUT> userFunction) {
+        super(userFunction);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<OUT>> output) {
+        super.setup(containingTask, config, output);
+        ClassLoader userCodeClassLoader = containingTask.getUserCodeClassLoader();
+        MemoryManager memoryManager = containingTask.getEnvironment().getMemoryManager();
+        IOManager ioManager = containingTask.getEnvironment().getIOManager();
+
+        keySelector = (KeySelector<IN, KEY>) config.getStatePartitioner(0, userCodeClassLoader);
+        keySerializer = config.getStateKeySerializer(userCodeClassLoader);
+        int keyLength = keySerializer.getLength();
+
+        TypeSerializer<IN> typeSerializerA = config.getTypeSerializerIn(0, userCodeClassLoader);
+        keyAndValueSerializer = new KeyAndValueSerializer<>(typeSerializerA, keyLength);
+
+        if (keyLength > 0) {
+            dataOutputSerializer = new DataOutputSerializer(keyLength);
+            comparator = new FixedLengthByteKeyComparator<>(keyLength);
+        } else {
+            dataOutputSerializer = new DataOutputSerializer(64);
+            comparator = new VariableLengthByteKeyComparator<>();
+        }
+
+        ExecutionConfig executionConfig = containingTask.getEnvironment().getExecutionConfig();
+
+        double managedMemoryFraction =
+                config.getManagedMemoryFractionOperatorUseCaseOfSlot(
+                        ManagedMemoryUseCase.OPERATOR,
+                        containingTask.getEnvironment().getTaskConfiguration(),
+                        userCodeClassLoader);
+
+        Configuration jobConfiguration = containingTask.getEnvironment().getJobConfiguration();
+
+        try {
+            sorter =
+                    ExternalSorter.newBuilder(
+                                    memoryManager,
+                                    containingTask,
+                                    keyAndValueSerializer,
+                                    comparator,
+                                    executionConfig)
+                            .memoryFraction(managedMemoryFraction)
+                            .enableSpilling(
+                                    ioManager,
+                                    jobConfiguration.get(AlgorithmOptions.SORT_SPILLING_THRESHOLD))
+                            .maxNumFileHandles(
+                                    jobConfiguration.get(AlgorithmOptions.SPILLING_MAX_FAN) / 2)
+                            .objectReuse(executionConfig.isObjectReuseEnabled())
+                            .largeRecords(
+                                    jobConfiguration.get(
+                                            AlgorithmOptions.USE_LARGE_RECORDS_HANDLER))
+                            .build();
+        } catch (MemoryAllocationException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    @Override
+    public void endInput(int inputId) throws Exception {
+        if (inputId == 1) {
+            sorter.finishReading();
+            remainingInputNum--;
+        } else {
+            throw new RuntimeException("Unknown inputId " + inputId);
+        }
+
+        if (remainingInputNum > 0) {

Review Comment:
   If it is impossible to have `remainingInputNum > 0` here, it would be simpler to remove this check.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/OneInputTransformationTranslator.java:
##########
@@ -50,24 +50,47 @@ public Collection<Integer> translateForBatchInternal(
                         keySelector,
                         transformation.getStateKeyType(),
                         context);
-        boolean isKeyed = keySelector != null;
-        if (isKeyed) {
-            BatchExecutionUtils.applyBatchExecutionSettings(
-                    transformation.getId(), context, StreamConfig.InputRequirement.SORTED);
-        }
+
+        oneInputTransformationOutputEOFSetting(transformation, context);
 
         return ids;
     }
 
     @Override
     public Collection<Integer> translateForStreamingInternal(
             final OneInputTransformation<IN, OUT> transformation, final Context context) {
-        return translateInternal(
-                transformation,
-                transformation.getOperatorFactory(),
-                transformation.getInputType(),
-                transformation.getStateKeySelector(),
-                transformation.getStateKeyType(),
-                context);
+        Collection<Integer> ids =
+                translateInternal(
+                        transformation,
+                        transformation.getOperatorFactory(),
+                        transformation.getInputType(),
+                        transformation.getStateKeySelector(),
+                        transformation.getStateKeyType(),
+                        context);
+
+        if (transformation.isOutputOnEOF()) {
+            // Try to apply batch execution settings for streaming mode transformation.
+            oneInputTransformationOutputEOFSetting(transformation, context);
+        }
+
+        return ids;
+    }
+
+    private void oneInputTransformationOutputEOFSetting(

Review Comment:
   It might be a bit confusing to have `OutputEOF` in the method's name because this method is invoked under batch mode even if the operator does not have `isOutputOnEOF = true`.
   
   How about naming this method `maybeApplyBatchExecutionSettings`?



##########
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkTransformationTranslatorITCase.java:
##########
@@ -225,7 +225,8 @@ private StreamNode findWriter(StreamGraph streamGraph) {
     }
 
     private StreamNode findCommitter(StreamGraph streamGraph) {
-        return findNodeName(streamGraph, name -> name.contains("Committer"));
+        return findNodeName(

Review Comment:
   Can you explain why we need this change?



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/EOFAggregationOperator.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.sort;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.AlgorithmOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.core.memory.ManagedMemoryUseCase;
+import org.apache.flink.runtime.io.disk.iomanager.IOManager;
+import org.apache.flink.runtime.memory.MemoryAllocationException;
+import org.apache.flink.runtime.memory.MemoryManager;
+import org.apache.flink.runtime.operators.sort.ExternalSorter;
+import org.apache.flink.runtime.operators.sort.PushSorter;
+import org.apache.flink.runtime.util.NonReusingKeyGroupedIterator;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.MutableObjectIterator;
+
+/**
+ * This operator would replace the regular Aggregate process when the WindowAssigner is
+ * EndOfStreamWindows.
+ */
+@Internal
+public class EOFAggregationOperator<IN, KEY, ACC, OUT>
+        extends AbstractUdfStreamOperator<OUT, AggregateFunction<IN, ACC, OUT>>
+        implements OneInputStreamOperator<IN, OUT>, BoundedMultiInput {
+
+    private PushSorter<Tuple2<byte[], StreamRecord<IN>>> sorter;
+    TypeSerializer<Tuple2<byte[], StreamRecord<IN>>> keyAndValueSerializer;
+    TypeComparator<Tuple2<byte[], StreamRecord<IN>>> comparator;
+    private KeySelector<IN, KEY> keySelector;
+    private TypeSerializer<KEY> keySerializer;
+    private DataOutputSerializer dataOutputSerializer;
+    private long lastWatermarkTimestamp = Long.MIN_VALUE;
+    private int remainingInputNum = 1;
+    private transient OperatorAttributes operatorAttributes;
+
+    public EOFAggregationOperator(AggregateFunction<IN, ACC, OUT> userFunction) {
+        super(userFunction);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<OUT>> output) {
+        super.setup(containingTask, config, output);
+        ClassLoader userCodeClassLoader = containingTask.getUserCodeClassLoader();
+        MemoryManager memoryManager = containingTask.getEnvironment().getMemoryManager();
+        IOManager ioManager = containingTask.getEnvironment().getIOManager();
+
+        keySelector = (KeySelector<IN, KEY>) config.getStatePartitioner(0, userCodeClassLoader);
+        keySerializer = config.getStateKeySerializer(userCodeClassLoader);
+        int keyLength = keySerializer.getLength();
+
+        TypeSerializer<IN> typeSerializerA = config.getTypeSerializerIn(0, userCodeClassLoader);
+        keyAndValueSerializer = new KeyAndValueSerializer<>(typeSerializerA, keyLength);
+
+        if (keyLength > 0) {
+            dataOutputSerializer = new DataOutputSerializer(keyLength);
+            comparator = new FixedLengthByteKeyComparator<>(keyLength);
+        } else {
+            dataOutputSerializer = new DataOutputSerializer(64);
+            comparator = new VariableLengthByteKeyComparator<>();
+        }
+
+        ExecutionConfig executionConfig = containingTask.getEnvironment().getExecutionConfig();
+
+        double managedMemoryFraction =
+                config.getManagedMemoryFractionOperatorUseCaseOfSlot(
+                        ManagedMemoryUseCase.OPERATOR,
+                        containingTask.getEnvironment().getTaskConfiguration(),
+                        userCodeClassLoader);
+
+        Configuration jobConfiguration = containingTask.getEnvironment().getJobConfiguration();
+
+        try {
+            sorter =
+                    ExternalSorter.newBuilder(
+                                    memoryManager,
+                                    containingTask,
+                                    keyAndValueSerializer,
+                                    comparator,
+                                    executionConfig)
+                            .memoryFraction(managedMemoryFraction)
+                            .enableSpilling(
+                                    ioManager,
+                                    jobConfiguration.get(AlgorithmOptions.SORT_SPILLING_THRESHOLD))
+                            .maxNumFileHandles(
+                                    jobConfiguration.get(AlgorithmOptions.SPILLING_MAX_FAN) / 2)

Review Comment:
   Given that there is only one sorter used in this class, we might not need to divide it by 2.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:
##########
@@ -1472,15 +1498,35 @@ private ResultPartitionType getResultPartitionType(StreamEdge edge) {
             case HYBRID_SELECTIVE:
                 return ResultPartitionType.HYBRID_SELECTIVE;
             case UNDEFINED:
-                return determineUndefinedResultPartitionType(edge.getPartitioner());
+                return determineUndefinedResultPartitionType(edge);
             default:
                 throw new UnsupportedOperationException(
                         "Data exchange mode " + edge.getExchangeMode() + " is not supported yet.");
         }
     }
 
-    private ResultPartitionType determineUndefinedResultPartitionType(
-            StreamPartitioner<?> partitioner) {
+    private ResultPartitionType determineUndefinedResultPartitionType(StreamEdge edge) {
+        if (outputBlockingNodesID.contains(edge.getSourceId())) {
+            // use corresponding partition type if the upstream node outputs EOF
+            edge.setBufferTimeout(ExecutionOptions.DISABLED_NETWORK_BUFFER_TIMEOUT);

Review Comment:
   Suppose operatorA and operatorB is not chained, operatorA sends records to operatorB, and operatorB has outputOnEOF = true, and execution mode is streaming. Will this PR guarantee that the edge between operatorA and operatorB is blocking (or hybrid shuffle), and operatorA runs in batch mode?
   
   If yes, can you provide reference to the code that achieves this? The code in this class seems to only guarantee that those operators chained with an operator with isOutputOnEOF = true will run in batch mode.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/EOFAggregationOperator.java:
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.sort;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.AlgorithmOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.core.memory.ManagedMemoryUseCase;
+import org.apache.flink.runtime.io.disk.iomanager.IOManager;
+import org.apache.flink.runtime.memory.MemoryAllocationException;
+import org.apache.flink.runtime.memory.MemoryManager;
+import org.apache.flink.runtime.operators.sort.ExternalSorter;
+import org.apache.flink.runtime.operators.sort.PushSorter;
+import org.apache.flink.runtime.util.NonReusingKeyGroupedIterator;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedMultiInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.MutableObjectIterator;
+
+/**
+ * This operator would replace the regular Aggregate process when the WindowAssigner is
+ * EndOfStreamWindows.
+ */
+@Internal
+public class EOFAggregationOperator<IN, KEY, ACC, OUT>
+        extends AbstractUdfStreamOperator<OUT, AggregateFunction<IN, ACC, OUT>>
+        implements OneInputStreamOperator<IN, OUT>, BoundedMultiInput {
+
+    private PushSorter<Tuple2<byte[], StreamRecord<IN>>> sorter;
+    TypeSerializer<Tuple2<byte[], StreamRecord<IN>>> keyAndValueSerializer;
+    TypeComparator<Tuple2<byte[], StreamRecord<IN>>> comparator;
+    private KeySelector<IN, KEY> keySelector;
+    private TypeSerializer<KEY> keySerializer;
+    private DataOutputSerializer dataOutputSerializer;
+    private long lastWatermarkTimestamp = Long.MIN_VALUE;
+    private int remainingInputNum = 1;
+    private transient OperatorAttributes operatorAttributes;
+
+    public EOFAggregationOperator(AggregateFunction<IN, ACC, OUT> userFunction) {
+        super(userFunction);
+    }
+
+    @Override
+    public void setup(
+            StreamTask<?, ?> containingTask,
+            StreamConfig config,
+            Output<StreamRecord<OUT>> output) {
+        super.setup(containingTask, config, output);
+        ClassLoader userCodeClassLoader = containingTask.getUserCodeClassLoader();
+        MemoryManager memoryManager = containingTask.getEnvironment().getMemoryManager();
+        IOManager ioManager = containingTask.getEnvironment().getIOManager();
+
+        keySelector = (KeySelector<IN, KEY>) config.getStatePartitioner(0, userCodeClassLoader);
+        keySerializer = config.getStateKeySerializer(userCodeClassLoader);
+        int keyLength = keySerializer.getLength();
+
+        TypeSerializer<IN> typeSerializerA = config.getTypeSerializerIn(0, userCodeClassLoader);
+        keyAndValueSerializer = new KeyAndValueSerializer<>(typeSerializerA, keyLength);
+
+        if (keyLength > 0) {
+            dataOutputSerializer = new DataOutputSerializer(keyLength);
+            comparator = new FixedLengthByteKeyComparator<>(keyLength);
+        } else {
+            dataOutputSerializer = new DataOutputSerializer(64);
+            comparator = new VariableLengthByteKeyComparator<>();
+        }
+
+        ExecutionConfig executionConfig = containingTask.getEnvironment().getExecutionConfig();
+
+        double managedMemoryFraction =
+                config.getManagedMemoryFractionOperatorUseCaseOfSlot(
+                        ManagedMemoryUseCase.OPERATOR,
+                        containingTask.getEnvironment().getTaskConfiguration(),
+                        userCodeClassLoader);
+
+        Configuration jobConfiguration = containingTask.getEnvironment().getJobConfiguration();
+
+        try {
+            sorter =
+                    ExternalSorter.newBuilder(
+                                    memoryManager,
+                                    containingTask,
+                                    keyAndValueSerializer,
+                                    comparator,
+                                    executionConfig)
+                            .memoryFraction(managedMemoryFraction)
+                            .enableSpilling(
+                                    ioManager,
+                                    jobConfiguration.get(AlgorithmOptions.SORT_SPILLING_THRESHOLD))
+                            .maxNumFileHandles(
+                                    jobConfiguration.get(AlgorithmOptions.SPILLING_MAX_FAN) / 2)
+                            .objectReuse(executionConfig.isObjectReuseEnabled())
+                            .largeRecords(
+                                    jobConfiguration.get(
+                                            AlgorithmOptions.USE_LARGE_RECORDS_HANDLER))
+                            .build();
+        } catch (MemoryAllocationException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    @Override
+    public void endInput(int inputId) throws Exception {
+        if (inputId == 1) {
+            sorter.finishReading();
+            remainingInputNum--;
+        } else {
+            throw new RuntimeException("Unknown inputId " + inputId);
+        }
+
+        if (remainingInputNum > 0) {
+            return;
+        }
+
+        MutableObjectIterator<Tuple2<byte[], StreamRecord<IN>>> iterator = sorter.getIterator();
+        TimestampedCollector<OUT> outputCollector = new TimestampedCollector<>(output);
+        NonReusingKeyGroupedIterator<Tuple2<byte[], StreamRecord<IN>>> keyGroupedIterator =
+                new NonReusingKeyGroupedIterator<>(iterator, comparator);
+
+        while (keyGroupedIterator.nextKey()) {
+            ACC acc = userFunction.createAccumulator();
+            NonReusingKeyGroupedIterator<Tuple2<byte[], StreamRecord<IN>>>.ValuesIterator records =
+                    keyGroupedIterator.getValues();
+            for (Tuple2<byte[], StreamRecord<IN>> record : records) {
+                acc = userFunction.add(record.f1.getValue(), acc);
+            }
+            outputCollector.collect(userFunction.getResult(acc));
+        }
+
+        Watermark watermark = new Watermark(lastWatermarkTimestamp);
+        if (getTimeServiceManager().isPresent()) {
+            getTimeServiceManager().get().advanceWatermark(watermark);
+        }
+        output.emitWatermark(watermark);
+    }
+
+    @Override
+    public void processWatermark(Watermark watermark) throws Exception {
+        if (lastWatermarkTimestamp > watermark.getTimestamp()) {
+            throw new RuntimeException("Invalid watermark");
+        }
+        lastWatermarkTimestamp = watermark.getTimestamp();
+    }
+
+    @Override
+    public void close() throws Exception {
+        super.close();
+        sorter.close();
+    }
+
+    @Override
+    public void processElement(StreamRecord<IN> streamRecord) throws Exception {
+        KEY key = keySelector.getKey(streamRecord.getValue());
+        keySerializer.serialize(key, dataOutputSerializer);
+        byte[] serializedKey = dataOutputSerializer.getCopyOfBuffer();
+        dataOutputSerializer.clear();
+        sorter.writeRecord(Tuple2.of(serializedKey, streamRecord));
+    }
+
+    @Override
+    public OperatorAttributes getOperatorAttributes() {

Review Comment:
   It might be simpler to just instantiate and return `OperatorAttributes` without caching it. The reason is that JM should not have to repeatedly invoke `getOperatorAttributes()` for the same operator instance.
   
   And JM does need to repeatedly invoke `getOperatorAttributes()` for the same instance, it might be simpler to cache it in Flink infra (e.g. JM, OneInputTransformation, AbstractMultipleInputTransformation) instead of having each operator (which is written by operator developer) maintaining this cache.
   
   Same for `EOFCoGroupOperator`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1307046686


##########
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamGraphGeneratorTest.java:
##########
@@ -960,6 +962,51 @@ public void testCacheSideOutput() {
         verifyCacheConsumeNode(env, upstreamParallelism, cacheTransformation);
     }
 
+    @Test
+    public void testTransformOutputEOFOperator() {

Review Comment:
   Thanks for your correction.  Two related tests are removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on pull request #23000: [FLINK-32594][runtime] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on PR #23000:
URL: https://github.com/apache/flink/pull/23000#issuecomment-1702377438

   @lindong28 could you please take another look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1306883153


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/OneInputTransformationTranslator.java:
##########
@@ -50,24 +50,47 @@ public Collection<Integer> translateForBatchInternal(
                         keySelector,
                         transformation.getStateKeyType(),
                         context);
-        boolean isKeyed = keySelector != null;
-        if (isKeyed) {
-            BatchExecutionUtils.applyBatchExecutionSettings(
-                    transformation.getId(), context, StreamConfig.InputRequirement.SORTED);
-        }
+
+        oneInputTransformationOutputEOFSetting(transformation, context);
 
         return ids;
     }
 
     @Override
     public Collection<Integer> translateForStreamingInternal(
             final OneInputTransformation<IN, OUT> transformation, final Context context) {
-        return translateInternal(
-                transformation,
-                transformation.getOperatorFactory(),
-                transformation.getInputType(),
-                transformation.getStateKeySelector(),
-                transformation.getStateKeyType(),
-                context);
+        Collection<Integer> ids =
+                translateInternal(
+                        transformation,
+                        transformation.getOperatorFactory(),
+                        transformation.getInputType(),
+                        transformation.getStateKeySelector(),
+                        transformation.getStateKeyType(),
+                        context);
+
+        if (transformation.isOutputOnEOF()) {
+            // Try to apply batch execution settings for streaming mode transformation.
+            oneInputTransformationOutputEOFSetting(transformation, context);
+        }
+
+        return ids;
+    }
+
+    private void oneInputTransformationOutputEOFSetting(

Review Comment:
   I agree, and also modified in `TwoInputTransformationTranslator` and `MultiInputTransformationTranslator`. Thanks for your suggestion.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/OneInputTransformationTranslator.java:
##########
@@ -50,24 +50,47 @@ public Collection<Integer> translateForBatchInternal(
                         keySelector,
                         transformation.getStateKeyType(),
                         context);
-        boolean isKeyed = keySelector != null;
-        if (isKeyed) {
-            BatchExecutionUtils.applyBatchExecutionSettings(
-                    transformation.getId(), context, StreamConfig.InputRequirement.SORTED);
-        }
+
+        oneInputTransformationOutputEOFSetting(transformation, context);
 
         return ids;
     }
 
     @Override
     public Collection<Integer> translateForStreamingInternal(
             final OneInputTransformation<IN, OUT> transformation, final Context context) {
-        return translateInternal(
-                transformation,
-                transformation.getOperatorFactory(),
-                transformation.getInputType(),
-                transformation.getStateKeySelector(),
-                transformation.getStateKeyType(),
-                context);
+        Collection<Integer> ids =
+                translateInternal(
+                        transformation,
+                        transformation.getOperatorFactory(),
+                        transformation.getInputType(),
+                        transformation.getStateKeySelector(),
+                        transformation.getStateKeyType(),
+                        context);
+
+        if (transformation.isOutputOnEOF()) {
+            // Try to apply batch execution settings for streaming mode transformation.
+            oneInputTransformationOutputEOFSetting(transformation, context);
+        }
+
+        return ids;
+    }
+
+    private void oneInputTransformationOutputEOFSetting(

Review Comment:
   I agree, and also modified in `TwoInputTransformationTranslator` and `MultiInputTransformationTranslator`. Thanks for your suggestion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1268169556


##########
flink-core/src/main/java/org/apache/flink/api/common/functions/Function.java:
##########
@@ -27,4 +28,16 @@
  * method) interfaces that can be implemented via Java 8 lambdas.
  */
 @Public
-public interface Function extends java.io.Serializable {}
+public interface Function extends java.io.Serializable {
+    /** Returns true iff the operator can only emit records after inputs have reached EOF. */
+    @Internal
+    default boolean isOutputEOF() {

Review Comment:
   It's more reasonable to overwrite `OperatorAttribute` in an operator, now you can see the new usage in  `StreamingJobGraphGeneratorTest#testOverwriteOperatorAttributesPartitionTypes`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Sxnan commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "Sxnan (via GitHub)" <gi...@apache.org>.
Sxnan commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1266328740


##########
flink-core/src/main/java/org/apache/flink/api/dag/Transformation.java:
##########
@@ -637,4 +637,16 @@ public int hashCode() {
         result = 31 * result + (int) (bufferTimeout ^ (bufferTimeout >>> 32));
         return result;
     }
+

Review Comment:
   Would it make more sense to add these methods to the `PhysicalTransformation`?  These methods don't make sense for non-physical transformation, where it does not correspond to a physical operation at runtime, such as union, partitioning, etc. 



##########
flink-core/src/main/java/org/apache/flink/api/common/functions/Function.java:
##########
@@ -27,4 +28,16 @@
  * method) interfaces that can be implemented via Java 8 lambdas.
  */
 @Public
-public interface Function extends java.io.Serializable {}
+public interface Function extends java.io.Serializable {

Review Comment:
   I think we don't need to have the `isOutputEOF` and `internalSOrterSupported` for the `Function` interface, as the function is not able to emit records after the inputs reach EOF or internally sort the inputs by key, which means that these two methods always return false.
   
   The `isOutputEOF` and `internalSorterSupported` can be determined by the `AbstractUdfStreamOperator` itself.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorAttributesBuilder.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.Optional;
+
+/** The builder class for {@link OperatorAttributes}. */
+@PublicEvolving
+public class OperatorAttributesBuilder {
+
+    private static final Logger LOG = LoggerFactory.getLogger(StreamingJobGraphGenerator.class);
+    @Nullable private Boolean outputOnEOF = null;
+    @Nullable private Boolean outputOnCheckpoint = null;
+    @Nullable private Boolean internalSorterSupported = null;
+
+    public OperatorAttributesBuilder() {}
+
+    public OperatorAttributesBuilder setOutputOnEOF(boolean outputOnEOF) {
+        this.outputOnEOF = outputOnEOF;
+        return this;
+    }
+
+    public OperatorAttributesBuilder setOutputOnCheckpoint(boolean outputOnCheckpoint) {
+        this.outputOnCheckpoint = outputOnCheckpoint;
+        return this;
+    }
+
+    public OperatorAttributesBuilder setInternalSorterSupported(boolean internalSorterSupported) {
+        this.internalSorterSupported = internalSorterSupported;
+        return this;
+    }
+
+    /**
+     * If any operator attribute is null, we will use the following default values. - outputOnEOF

Review Comment:
   Update the comment according to the FLIP. And we can log the null value at DEBUG level.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:
##########
@@ -702,6 +715,12 @@ private List<StreamEdge> createChain(
                                 (k) -> chainInfo.newChain(nonChainable.getTargetId())),
                         chainEntryPoints);
             }
+            for (StreamEdge edge : currentNode.getOutEdges()) {

Review Comment:
   If the currentNode has two downstream nodes, one is outputEOF and one is not. Should we still mark the currentNode as outputEOF? I think naively marking the currentNode as outputEOF will effectively make the non-outputEOF to only output on EOF, which seems undesired. 



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java:
##########
@@ -649,4 +654,12 @@ public OperatorID getOperatorID() {
     protected Optional<InternalTimeServiceManager<?>> getTimeServiceManager() {
         return Optional.ofNullable(timeServiceManager);
     }
+
+    @Override
+    public OperatorAttributes getOperatorAttributes() {
+        if (operatorAttributes == null) {

Review Comment:
   I think it is unnecessary to overwrite the method in `AbstractStreamOperator`, given that the implementation is same as the `StreamOperator`



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:
##########
@@ -1488,6 +1507,11 @@ private ResultPartitionType determineUndefinedResultPartitionType(
                     return ResultPartitionType.BLOCKING;
                 }
             case ALL_EDGES_PIPELINED:
+                if (outputEOFVerticesID.contains(edge.getSourceId())) {

Review Comment:
   I think we should mark the partition type blocking if the operator outputs EOF for all global stream exchange mode.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:
##########
@@ -184,6 +184,9 @@ public static JobGraph createJobGraph(
 
     private final Map<Integer, InputOutputFormatContainer> chainedInputOutputFormats;
 
+    // The node id would be in this Set if it is an upstream node of an outputEOF node.

Review Comment:
   Maybe simpler to say: The ids of the nodes that output on EOF.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorAttributesBuilder.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.Optional;
+
+/** The builder class for {@link OperatorAttributes}. */
+@PublicEvolving
+public class OperatorAttributesBuilder {
+
+    private static final Logger LOG = LoggerFactory.getLogger(StreamingJobGraphGenerator.class);
+    @Nullable private Boolean outputOnEOF = null;
+    @Nullable private Boolean outputOnCheckpoint = null;
+    @Nullable private Boolean internalSorterSupported = null;
+
+    public OperatorAttributesBuilder() {}
+
+    public OperatorAttributesBuilder setOutputOnEOF(boolean outputOnEOF) {
+        this.outputOnEOF = outputOnEOF;
+        return this;
+    }
+
+    public OperatorAttributesBuilder setOutputOnCheckpoint(boolean outputOnCheckpoint) {
+        this.outputOnCheckpoint = outputOnCheckpoint;
+        return this;
+    }
+
+    public OperatorAttributesBuilder setInternalSorterSupported(boolean internalSorterSupported) {
+        this.internalSorterSupported = internalSorterSupported;
+        return this;
+    }
+
+    /**
+     * If any operator attribute is null, we will use the following default values. - outputOnEOF

Review Comment:
   Nit: format the comment



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:
##########
@@ -1488,6 +1507,11 @@ private ResultPartitionType determineUndefinedResultPartitionType(
                     return ResultPartitionType.BLOCKING;
                 }
             case ALL_EDGES_PIPELINED:
+                if (outputEOFVerticesID.contains(edge.getSourceId())) {
+                    // Blocking the edge when the node is an outputEOF node or its upstream node;

Review Comment:
   How about this:
   Use blocking partition type if the upstream node outputs EOF.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/OperatorAttributes.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.runtime.event.RuntimeEvent;
+
+import java.io.IOException;
+
+/**
+ * OperatorAttributes element provides Job Manager with information that can be used to optimize job
+ * performance.
+ */
+@PublicEvolving
+public class OperatorAttributes extends RuntimeEvent {

Review Comment:
   OperatorAtrributes should not extend RuntimeEvent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on PR #23000:
URL: https://github.com/apache/flink/pull/23000#issuecomment-1648855116

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1267862410


##########
flink-core/src/main/java/org/apache/flink/api/common/functions/Function.java:
##########
@@ -27,4 +28,16 @@
  * method) interfaces that can be implemented via Java 8 lambdas.
  */
 @Public
-public interface Function extends java.io.Serializable {}
+public interface Function extends java.io.Serializable {
+    /** Returns true iff the operator can only emit records after inputs have reached EOF. */
+    @Internal
+    default boolean isOutputEOF() {

Review Comment:
   Thanks for your comments! I simply want to provide a method for user defined function to determine the OperatorAttribute directly. Indeed, it does not make sense to connect `Function` and `OperatorAttribute` together. I will modify this later. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RanJinh commented on a diff in pull request #23000: [FLINK-32594][runtime] Use blocking ResultPartitionType if operator only outputs records on EOF

Posted by "RanJinh (via GitHub)" <gi...@apache.org>.
RanJinh commented on code in PR #23000:
URL: https://github.com/apache/flink/pull/23000#discussion_r1267868354


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:
##########
@@ -184,6 +184,9 @@ public static JobGraph createJobGraph(
 
     private final Map<Integer, InputOutputFormatContainer> chainedInputOutputFormats;
 
+    // The node id would be in this Set if it is an upstream node of an outputEOF node.

Review Comment:
   Thanks for your detailed comments! I will check all of the code comments according to your suggestion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org