You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ar...@apache.org on 2024/12/03 09:16:21 UTC

(flink) branch release-1.20 updated (71fd38037cd -> 504c866b347)

This is an automated email from the ASF dual-hosted git repository.

arvid pushed a change to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git


    from 71fd38037cd [FLINK-36643][filesystems] Update aws-java-sdk-core to 1.12.779
     new a613d2c2466 [FLINK-36379] Refactor sink test assertions
     new e62c7ab8dca [FLINK-36379] Optimize global committers
     new 67932dc1f49 [FLINK-36379] Optimize committers with UC disabled
     new 504c866b347 [FLINK-36788] Fix GlobalCommitter expansion

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../file/sink/compactor/CompactorOperatorTest.java | 107 ++++++-------
 .../connector/sink2/GlobalCommitterOperator.java   | 136 ++++++++++++-----
 .../connector/sink2/StandardSinkTopologies.java    |  23 +--
 .../connector/sink2/SupportsPreCommitTopology.java |  13 ++
 .../streaming/api/graph/StreamGraphGenerator.java  |   3 +
 .../transformations/GlobalCommitterTransform.java  |  84 +++++++++++
 .../runtime/operators/sink/CommitterOperator.java  |   9 +-
 .../committables/CheckpointCommittableManager.java |  10 +-
 .../CheckpointCommittableManagerImpl.java          |  39 ++++-
 .../sink/committables/CommittableCollector.java    |   8 +-
 .../GlobalCommitterTransformationTranslator.java   | 140 +++++++++++++++++
 .../translators/SinkTransformationTranslator.java  |  18 ++-
 .../connector/sink2/CommittableSummaryAssert.java  |  18 +--
 ...sert.java => CommittableWithLineageAssert.java} |  15 +-
 .../sink2/GlobalCommitterOperatorTest.java         | 143 ++++++++++++------
 .../api/connector/sink2/SinkV2Assertions.java      |  28 +++-
 .../SinkTransformationTranslatorITCaseBase.java    |  63 ++++++++
 .../SinkV1TransformationTranslatorITCase.java      |  64 ++------
 ...V2TransformationTranslatorDeprecatedITCase.java |  14 ++
 .../SinkV2TransformationTranslatorITCase.java      |  14 ++
 .../operators/sink/CommitterOperatorTestBase.java  |  75 +++++-----
 .../runtime/operators/sink/SinkTestUtil.java       |  64 --------
 .../operators/sink/SinkWriterOperatorTestBase.java | 166 ++++++++-------------
 .../runtime/operators/sink/TestSinkV2.java         |  15 +-
 .../CheckpointCommittableManagerImplTest.java      |  13 +-
 .../CommittableCollectorSerializerTest.java        |  64 +++-----
 .../committables/CommittableCollectorTest.java     |  11 +-
 .../SubtaskCommittableManagerTest.java             |  22 ++-
 .../operators/sink/deprecated/TestSinkV2.java      |   4 +-
 29 files changed, 861 insertions(+), 522 deletions(-)
 create mode 100644 flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/GlobalCommitterTransform.java
 create mode 100644 flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/GlobalCommitterTransformationTranslator.java
 rename flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/{CommittableWithLinageAssert.java => CommittableWithLineageAssert.java} (70%)


(flink) 02/04: [FLINK-36379] Optimize global committers

Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

arvid pushed a commit to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git

commit e62c7ab8dca92b2dd285641a726b2869de2a126a
Author: Arvid Heise <ar...@apache.org>
AuthorDate: Wed Oct 2 16:15:25 2024 +0200

    [FLINK-36379] Optimize global committers
    
    Global committers used to trail a full checkpoint behind committer. That means that data appeared only after >2*checkpoint interval in the sinks that use it (e.g. delta). However, committables in the global committers are already part of the first checkpoint and are idempotent: On recovery, they are resend from the committer to the global committer. Thus, the global committer can actually be seen as stateless and doesn't need to conduct its own 2PC protocol.
    
    This commit lets the global committer collect all committables on input (as before) but immediately tries to commit when it has received all (deducible from CommitterSummary - which was always the original intent of that message). Thus, in most cases, GlobalCommitter ignores notifyCheckpointCompleted now as the state of the checkpoint can be inferred by received committables from upstream.
    
    There are special cases where a global committer is directly chained to a writer. In this case, the global committer does need to conduct a 2PC protocol in place of the committer. To differentiate these cases, the global committer now has its own transformation.
    
    (cherry picked from commit 67be29a003dfcaa1caccee716e6860efc1ea194d)
---
 .../connector/sink2/GlobalCommitterOperator.java   | 136 +++++++++++++++-----
 .../connector/sink2/StandardSinkTopologies.java    |  23 +---
 .../connector/sink2/SupportsPreCommitTopology.java |  13 ++
 .../streaming/api/graph/StreamGraphGenerator.java  |   3 +
 .../transformations/GlobalCommitterTransform.java  |  84 ++++++++++++
 .../committables/CheckpointCommittableManager.java |  10 +-
 .../CheckpointCommittableManagerImpl.java          |  10 +-
 .../sink/committables/CommittableCollector.java    |   8 +-
 .../GlobalCommitterTransformationTranslator.java   | 129 +++++++++++++++++++
 .../sink2/GlobalCommitterOperatorTest.java         | 143 ++++++++++++++-------
 .../committables/CommittableCollectorTest.java     |  11 +-
 11 files changed, 462 insertions(+), 108 deletions(-)

diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/GlobalCommitterOperator.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/GlobalCommitterOperator.java
index 6c11b3ff022..b3195d286d6 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/GlobalCommitterOperator.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/GlobalCommitterOperator.java
@@ -18,6 +18,7 @@
 
 package org.apache.flink.streaming.api.connector.sink2;
 
+import org.apache.flink.annotation.Internal;
 import org.apache.flink.api.common.state.ListState;
 import org.apache.flink.api.common.state.ListStateDescriptor;
 import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
@@ -30,7 +31,6 @@ import org.apache.flink.runtime.state.StateInitializationContext;
 import org.apache.flink.runtime.state.StateSnapshotContext;
 import org.apache.flink.streaming.api.graph.StreamConfig;
 import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
-import org.apache.flink.streaming.api.operators.BoundedOneInput;
 import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
 import org.apache.flink.streaming.api.operators.Output;
 import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
@@ -49,11 +49,54 @@ import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
 
+import static org.apache.flink.streaming.api.connector.sink2.CommittableMessage.EOI;
 import static org.apache.flink.util.Preconditions.checkNotNull;
 import static org.apache.flink.util.Preconditions.checkState;
 
-class GlobalCommitterOperator<CommT, GlobalCommT> extends AbstractStreamOperator<Void>
-        implements OneInputStreamOperator<CommittableMessage<CommT>, Void>, BoundedOneInput {
+/**
+ * Implements the {@code GlobalCommitter}.
+ *
+ * <p>This operator usually trails behind a {@code CommitterOperator}. In this case, the global
+ * committer will receive committables from the committer operator through {@link
+ * #processElement(StreamRecord)}. Once all committables from all subtasks have been received, the
+ * global committer will commit them. This approach also works for any number of intermediate custom
+ * operators between the committer and the global committer in a custom post-commit topology.
+ *
+ * <p>That means that the global committer will not wait for {@link
+ * #notifyCheckpointComplete(long)}. In many cases, it receives the callback before the actual
+ * committables anyway. So it would effectively globally commit one checkpoint later.
+ *
+ * <p>However, we can leverage the following observation: the global committer will only receive
+ * committables iff the respective checkpoint was completed and upstream committers received the
+ * {@link #notifyCheckpointComplete(long)}. So by waiting for all committables of a given
+ * checkpoint, we implicitly know that the checkpoint was successful and the global committer is
+ * supposed to globally commit.
+ *
+ * <p>Note that committables of checkpoint X are not checkpointed in X because the global committer
+ * is trailing behind the checkpoint. They are replayed from the committer state in case of an
+ * error. The state only includes incomplete checkpoints coming from upstream committers not
+ * receiving {@link #notifyCheckpointComplete(long)}. All committables received are successful.
+ *
+ * <p>In rare cases, the GlobalCommitterOperator may not be connected (in)directly to a committer
+ * but instead is connected (in)directly to a writer. In this case, the global committer needs to
+ * perform the 2PC protocol instead of the committer. Thus, we absolutely need to use {@link
+ * #notifyCheckpointComplete(long)} similarly to the {@code CommitterOperator}. Hence, {@link
+ * #commitOnInput} is set to false in this case. In particular, the following three prerequisites
+ * must be met:
+ *
+ * <ul>
+ *   <li>No committer is upstream of which we could implicitly infer {@link
+ *       #notifyCheckpointComplete(long)} as sketched above.
+ *   <li>The application runs in streaming mode.
+ *   <li>Checkpointing is enabled.
+ * </ul>
+ *
+ * <p>In all other cases (batch or upstream committer or checkpointing is disabled), the global
+ * committer commits on input.
+ */
+@Internal
+public class GlobalCommitterOperator<CommT, GlobalCommT> extends AbstractStreamOperator<Void>
+        implements OneInputStreamOperator<CommittableMessage<CommT>, Void> {
 
     /** The operator's state descriptor. */
     private static final ListStateDescriptor<byte[]> GLOBAL_COMMITTER_OPERATOR_RAW_STATES_DESC =
@@ -63,6 +106,11 @@ class GlobalCommitterOperator<CommT, GlobalCommT> extends AbstractStreamOperator
     private final SerializableSupplier<Committer<CommT>> committerFactory;
     private final SerializableSupplier<SimpleVersionedSerializer<CommT>>
             committableSerializerFactory;
+    /**
+     * Depending on whether there is an upstream committer or it's connected to a writer, we may
+     * either wait for notifyCheckpointCompleted or not.
+     */
+    private final boolean commitOnInput;
 
     private ListState<GlobalCommittableWrapper<CommT, GlobalCommT>> globalCommitterState;
     private Committer<CommT> committer;
@@ -75,11 +123,13 @@ class GlobalCommitterOperator<CommT, GlobalCommT> extends AbstractStreamOperator
     @Nullable private SimpleVersionedSerializer<GlobalCommT> globalCommittableSerializer;
     private List<GlobalCommT> sinkV1State = new ArrayList<>();
 
-    GlobalCommitterOperator(
+    public GlobalCommitterOperator(
             SerializableSupplier<Committer<CommT>> committerFactory,
-            SerializableSupplier<SimpleVersionedSerializer<CommT>> committableSerializerFactory) {
+            SerializableSupplier<SimpleVersionedSerializer<CommT>> committableSerializerFactory,
+            boolean commitOnInput) {
         this.committerFactory = checkNotNull(committerFactory);
         this.committableSerializerFactory = checkNotNull(committableSerializerFactory);
+        this.commitOnInput = commitOnInput;
     }
 
     @Override
@@ -113,20 +163,11 @@ class GlobalCommitterOperator<CommT, GlobalCommT> extends AbstractStreamOperator
     @Override
     public void initializeState(StateInitializationContext context) throws Exception {
         super.initializeState(context);
-        final CommittableCollectorSerializer<CommT> committableCollectorSerializer =
-                new CommittableCollectorSerializer<>(
-                        committableSerializer,
-                        getRuntimeContext().getTaskInfo().getIndexOfThisSubtask(),
-                        getRuntimeContext().getTaskInfo().getMaxNumberOfParallelSubtasks(),
-                        metricGroup);
-        final SimpleVersionedSerializer<GlobalCommittableWrapper<CommT, GlobalCommT>> serializer =
-                new GlobalCommitterSerializer<>(
-                        committableCollectorSerializer, globalCommittableSerializer, metricGroup);
         globalCommitterState =
                 new SimpleVersionedListState<>(
                         context.getOperatorStateStore()
                                 .getListState(GLOBAL_COMMITTER_OPERATOR_RAW_STATES_DESC),
-                        serializer);
+                        getCommitterStateSerializer());
         if (context.isRestored()) {
             globalCommitterState
                     .get()
@@ -135,49 +176,74 @@ class GlobalCommitterOperator<CommT, GlobalCommT> extends AbstractStreamOperator
                                 sinkV1State.addAll(cc.getGlobalCommittables());
                                 committableCollector.merge(cc.getCommittableCollector());
                             });
-            lastCompletedCheckpointId = context.getRestoredCheckpointId().getAsLong();
             if (globalCommitter != null) {
                 sinkV1State = globalCommitter.filterRecoveredCommittables(sinkV1State);
             }
             // try to re-commit recovered transactions as quickly as possible
-            commit(lastCompletedCheckpointId);
+            if (context.getRestoredCheckpointId().isPresent()) {
+                commit(context.getRestoredCheckpointId().getAsLong());
+            }
         }
     }
 
+    private SimpleVersionedSerializer<GlobalCommittableWrapper<CommT, GlobalCommT>>
+            getCommitterStateSerializer() {
+        final CommittableCollectorSerializer<CommT> committableCollectorSerializer =
+                new CommittableCollectorSerializer<>(
+                        committableSerializer,
+                        getRuntimeContext().getTaskInfo().getIndexOfThisSubtask(),
+                        getRuntimeContext().getTaskInfo().getMaxNumberOfParallelSubtasks(),
+                        metricGroup);
+        return new GlobalCommitterSerializer<>(
+                committableCollectorSerializer, globalCommittableSerializer, metricGroup);
+    }
+
     @Override
     public void notifyCheckpointComplete(long checkpointId) throws Exception {
         super.notifyCheckpointComplete(checkpointId);
         checkState(
                 globalCommitter != null || sinkV1State.isEmpty(),
                 "GlobalCommitter is required to commit SinkV1 state.");
-        lastCompletedCheckpointId = Math.max(lastCompletedCheckpointId, checkpointId);
-        commit(lastCompletedCheckpointId);
+        if (!commitOnInput) {
+            commit(checkpointId);
+        }
     }
 
-    private void commit(long checkpointId) throws IOException, InterruptedException {
+    private void commit(long checkpointIdOrEOI) throws IOException, InterruptedException {
         if (globalCommitter != null && !sinkV1State.isEmpty()) {
             sinkV1State = globalCommitter.commit(sinkV1State);
         }
-        for (CheckpointCommittableManager<CommT> checkpoint :
-                committableCollector.getCheckpointCommittablesUpTo(checkpointId)) {
-            checkpoint.commit(committer);
-        }
-        committableCollector.compact();
-    }
 
-    @Override
-    public void endInput() throws Exception {
-        final CheckpointCommittableManager<CommT> endOfInputCommittable =
-                committableCollector.getEndOfInputCommittable();
-        if (endOfInputCommittable != null) {
-            do {
-                endOfInputCommittable.commit(committer);
-            } while (!committableCollector.isFinished());
-        }
+        lastCompletedCheckpointId = Math.max(lastCompletedCheckpointId, checkpointIdOrEOI);
+        // this is true for the last commit and we need to make sure that all committables are
+        // indeed committed as this function will never be invoked again
+        boolean waitForAllCommitted =
+                lastCompletedCheckpointId == EOI
+                        && committableCollector
+                                .getEndOfInputCommittable()
+                                .map(CheckpointCommittableManager::hasGloballyReceivedAll)
+                                .orElse(false);
+        do {
+            for (CheckpointCommittableManager<CommT> committable :
+                    committableCollector.getCheckpointCommittablesUpTo(lastCompletedCheckpointId)) {
+                if (committable.hasGloballyReceivedAll()) {
+                    committable.commit(committer);
+                }
+            }
+            committableCollector.compact();
+        } while (waitForAllCommitted && !committableCollector.isFinished());
     }
 
     @Override
     public void processElement(StreamRecord<CommittableMessage<CommT>> element) throws Exception {
         committableCollector.addMessage(element.getValue());
+
+        // commitOnInput implies that the global committer is not using notifyCheckpointComplete.
+        // Instead, it commits as soon as it receives all committables of a specific checkpoint.
+        // For commitOnInput=false, lastCompletedCheckpointId is only updated on
+        // notifyCheckpointComplete.
+        if (commitOnInput) {
+            commit(element.getValue().getCheckpointIdOrEOI());
+        }
     }
 }
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/StandardSinkTopologies.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/StandardSinkTopologies.java
index 956675c2b8e..691a797752e 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/StandardSinkTopologies.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/StandardSinkTopologies.java
@@ -19,12 +19,10 @@
 package org.apache.flink.streaming.api.connector.sink2;
 
 import org.apache.flink.annotation.Experimental;
-import org.apache.flink.api.common.typeinfo.Types;
 import org.apache.flink.api.connector.sink2.Committer;
 import org.apache.flink.core.io.SimpleVersionedSerializer;
 import org.apache.flink.streaming.api.datastream.DataStream;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-import org.apache.flink.streaming.api.transformations.PhysicalTransformation;
+import org.apache.flink.streaming.api.transformations.GlobalCommitterTransform;
 import org.apache.flink.util.function.SerializableSupplier;
 
 /** This utility class provides building blocks for custom topologies. */
@@ -43,19 +41,10 @@ public class StandardSinkTopologies {
             DataStream<CommittableMessage<CommT>> committables,
             SerializableSupplier<Committer<CommT>> committerFactory,
             SerializableSupplier<SimpleVersionedSerializer<CommT>> committableSerializer) {
-        final PhysicalTransformation<Void> transformation =
-                (PhysicalTransformation<Void>)
-                        committables
-                                .global()
-                                .transform(
-                                        GLOBAL_COMMITTER_TRANSFORMATION_NAME,
-                                        Types.VOID,
-                                        new GlobalCommitterOperator<>(
-                                                committerFactory, committableSerializer))
-                                .getTransformation();
-        transformation.setChainingStrategy(ChainingStrategy.ALWAYS);
-        transformation.setName(GLOBAL_COMMITTER_TRANSFORMATION_NAME);
-        transformation.setParallelism(1);
-        transformation.setMaxParallelism(1);
+        committables
+                .getExecutionEnvironment()
+                .addOperator(
+                        new GlobalCommitterTransform<>(
+                                committables, committerFactory, committableSerializer));
     }
 }
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/SupportsPreCommitTopology.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/SupportsPreCommitTopology.java
index 67f277b1b45..5470b92383a 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/SupportsPreCommitTopology.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/SupportsPreCommitTopology.java
@@ -19,6 +19,7 @@
 package org.apache.flink.streaming.api.connector.sink2;
 
 import org.apache.flink.annotation.Experimental;
+import org.apache.flink.api.common.functions.RuntimeContext;
 import org.apache.flink.api.connector.sink2.Committer;
 import org.apache.flink.api.connector.sink2.SinkWriter;
 import org.apache.flink.core.io.SimpleVersionedSerializer;
@@ -30,6 +31,18 @@ import org.apache.flink.streaming.api.datastream.DataStream;
  *
  * <p>It is recommended to use immutable committables because mutating committables can have
  * unexpected side-effects.
+ *
+ * <p>It's important that all {@link CommittableMessage}s are modified appropriately, such that all
+ * messages with the same subtask id will also be processed by the same {@link Committer} subtask
+ * and the {@link CommittableSummary} matches the respective count. If committables are combined or
+ * split in any way, the summary needs to be adjusted.
+ *
+ * <p>There is also no requirement to keep the subtask ids of the writer, they can be changed as
+ * long as there are no two summaries with the same subtask ids (and corresponding {@link
+ * CommittableWithLineage}). Subtask ids don't need to be consecutive or small. The global committer
+ * will use {@link CommittableSummary#getNumberOfSubtasks()} to determine if all committables have
+ * been received, so that number needs to correctly reflect the number of distinct subtask ids. The
+ * easiest way to guarantee all of this is to use {@link RuntimeContext#getTaskInfo()}.
  */
 @Experimental
 public interface SupportsPreCommitTopology<WriterResultT, CommittableT> {
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java
index 0e20431516e..2a64fabc711 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java
@@ -53,6 +53,7 @@ import org.apache.flink.streaming.api.transformations.BroadcastStateTransformati
 import org.apache.flink.streaming.api.transformations.CacheTransformation;
 import org.apache.flink.streaming.api.transformations.CoFeedbackTransformation;
 import org.apache.flink.streaming.api.transformations.FeedbackTransformation;
+import org.apache.flink.streaming.api.transformations.GlobalCommitterTransform;
 import org.apache.flink.streaming.api.transformations.KeyedBroadcastStateTransformation;
 import org.apache.flink.streaming.api.transformations.KeyedMultipleInputTransformation;
 import org.apache.flink.streaming.api.transformations.LegacySinkTransformation;
@@ -72,6 +73,7 @@ import org.apache.flink.streaming.api.transformations.UnionTransformation;
 import org.apache.flink.streaming.api.transformations.WithBoundedness;
 import org.apache.flink.streaming.runtime.translators.BroadcastStateTransformationTranslator;
 import org.apache.flink.streaming.runtime.translators.CacheTransformationTranslator;
+import org.apache.flink.streaming.runtime.translators.GlobalCommitterTransformationTranslator;
 import org.apache.flink.streaming.runtime.translators.KeyedBroadcastStateTransformationTranslator;
 import org.apache.flink.streaming.runtime.translators.LegacySinkTransformationTranslator;
 import org.apache.flink.streaming.runtime.translators.LegacySourceTransformationTranslator;
@@ -184,6 +186,7 @@ public class StreamGraphGenerator {
         tmp.put(KeyedMultipleInputTransformation.class, new MultiInputTransformationTranslator<>());
         tmp.put(SourceTransformation.class, new SourceTransformationTranslator<>());
         tmp.put(SinkTransformation.class, new SinkTransformationTranslator<>());
+        tmp.put(GlobalCommitterTransform.class, new GlobalCommitterTransformationTranslator<>());
         tmp.put(LegacySinkTransformation.class, new LegacySinkTransformationTranslator<>());
         tmp.put(LegacySourceTransformation.class, new LegacySourceTransformationTranslator<>());
         tmp.put(UnionTransformation.class, new UnionTransformationTranslator<>());
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/GlobalCommitterTransform.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/GlobalCommitterTransform.java
new file mode 100644
index 00000000000..a7dc311e896
--- /dev/null
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/GlobalCommitterTransform.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.connector.sink2.Committer;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.StandardSinkTopologies;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.util.function.SerializableSupplier;
+
+import org.apache.flink.shaded.guava31.com.google.common.collect.Lists;
+
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Transformation for global committer. Only used to fetch if the pipeline is streaming or batch
+ * with the respective {@link
+ * org.apache.flink.streaming.runtime.translators.GlobalCommitterTransformationTranslator}.
+ *
+ * @param <CommT>
+ */
+@Internal
+public class GlobalCommitterTransform<CommT> extends Transformation<Void> {
+
+    private final DataStream<CommittableMessage<CommT>> inputStream;
+    private final SerializableSupplier<Committer<CommT>> committerFactory;
+    private final SerializableSupplier<SimpleVersionedSerializer<CommT>> committableSerializer;
+
+    public GlobalCommitterTransform(
+            DataStream<CommittableMessage<CommT>> inputStream,
+            SerializableSupplier<Committer<CommT>> committerFactory,
+            SerializableSupplier<SimpleVersionedSerializer<CommT>> committableSerializer) {
+        super(StandardSinkTopologies.GLOBAL_COMMITTER_TRANSFORMATION_NAME, Types.VOID, 1, true);
+        this.inputStream = inputStream;
+        this.committerFactory = committerFactory;
+        this.committableSerializer = committableSerializer;
+    }
+
+    @Override
+    protected List<Transformation<?>> getTransitivePredecessorsInternal() {
+        final List<Transformation<?>> result = Lists.newArrayList();
+        result.add(this);
+        result.addAll(inputStream.getTransformation().getTransitivePredecessors());
+        return result;
+    }
+
+    @Override
+    public List<Transformation<?>> getInputs() {
+        return Collections.singletonList(inputStream.getTransformation());
+    }
+
+    public DataStream<CommittableMessage<CommT>> getInputStream() {
+        return inputStream;
+    }
+
+    public SerializableSupplier<Committer<CommT>> getCommitterFactory() {
+        return committerFactory;
+    }
+
+    public SerializableSupplier<SimpleVersionedSerializer<CommT>> getCommittableSerializer() {
+        return committableSerializer;
+    }
+}
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManager.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManager.java
index a4ce99f2bfe..ada7f8640f0 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManager.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManager.java
@@ -56,14 +56,20 @@ public interface CheckpointCommittableManager<CommT> {
      */
     CommittableSummary<CommT> getSummary(int emittingSubtaskId, int emittingNumberOfSubtasks);
 
+    boolean isFinished();
+
+    /**
+     * Returns true if all committables of all upstream subtasks arrived, which is only guaranteed
+     * to happen if the DOP of the caller is 1.
+     */
+    boolean hasGloballyReceivedAll();
+
     /**
      * Commits all due committables if all respective committables of the specific subtask and
      * checkpoint have been received.
      *
      * @param committer used to commit to the external system
      * @return successfully committed committables with meta information
-     * @throws IOException
-     * @throws InterruptedException
      */
     Collection<CommittableWithLineage<CommT>> commit(Committer<CommT> committer)
             throws IOException, InterruptedException;
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java
index 30e74c0e774..99d321d4cd1 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java
@@ -143,11 +143,19 @@ class CheckpointCommittableManagerImpl<CommT> implements CheckpointCommittableMa
                         .sum());
     }
 
-    boolean isFinished() {
+    @Override
+    public boolean isFinished() {
         return subtasksCommittableManagers.values().stream()
                 .allMatch(SubtaskCommittableManager::isFinished);
     }
 
+    @Override
+    public boolean hasGloballyReceivedAll() {
+        return subtasksCommittableManagers.size() == numberOfSubtasks
+                && subtasksCommittableManagers.values().stream()
+                        .allMatch(SubtaskCommittableManager::hasReceivedAll);
+    }
+
     @Override
     public Collection<CommittableWithLineage<CommT>> commit(Committer<CommT> committer)
             throws IOException, InterruptedException {
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
index 7a337059c04..37a7b72f7b3 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollector.java
@@ -26,14 +26,13 @@ import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
 import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
 import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
 
-import javax.annotation.Nullable;
-
 import java.util.Collection;
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
 import java.util.NavigableMap;
 import java.util.Objects;
+import java.util.Optional;
 import java.util.TreeMap;
 import java.util.stream.Collectors;
 
@@ -141,9 +140,8 @@ public class CommittableCollector<CommT> {
      *
      * @return {@link CheckpointCommittableManager}
      */
-    @Nullable
-    public CheckpointCommittableManager<CommT> getEndOfInputCommittable() {
-        return checkpointCommittables.get(EOI);
+    public Optional<CheckpointCommittableManager<CommT>> getEndOfInputCommittable() {
+        return Optional.ofNullable(checkpointCommittables.get(EOI));
     }
 
     /**
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/GlobalCommitterTransformationTranslator.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/GlobalCommitterTransformationTranslator.java
new file mode 100644
index 00000000000..69c4533b362
--- /dev/null
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/GlobalCommitterTransformationTranslator.java
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.translators;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.GlobalCommitterOperator;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.graph.TransformationTranslator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.GlobalCommitterTransform;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+import org.apache.flink.streaming.api.transformations.PhysicalTransformation;
+import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
+import org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperatorFactory;
+
+import java.util.ArrayDeque;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Queue;
+import java.util.Set;
+
+import static org.apache.flink.streaming.api.connector.sink2.StandardSinkTopologies.GLOBAL_COMMITTER_TRANSFORMATION_NAME;
+
+/**
+ * A {@link TransformationTranslator} for the {@link GlobalCommitterOperator}. The main purpose is
+ * to detect whether we set {@link GlobalCommitterOperator#commitOnInput} or not.
+ */
+@Internal
+public class GlobalCommitterTransformationTranslator<CommT>
+        implements TransformationTranslator<Void, GlobalCommitterTransform<CommT>> {
+
+    @Override
+    public Collection<Integer> translateForBatch(
+            GlobalCommitterTransform<CommT> transformation, Context context) {
+        return translateInternal(transformation, true);
+    }
+
+    @Override
+    public Collection<Integer> translateForStreaming(
+            GlobalCommitterTransform<CommT> transformation, Context context) {
+        return translateInternal(transformation, false);
+    }
+
+    private Collection<Integer> translateInternal(
+            GlobalCommitterTransform<CommT> globalCommitterTransform, boolean batch) {
+        DataStream<CommittableMessage<CommT>> inputStream =
+                globalCommitterTransform.getInputStream();
+        boolean checkpointingEnabled =
+                inputStream
+                        .getExecutionEnvironment()
+                        .getCheckpointConfig()
+                        .isCheckpointingEnabled();
+        boolean commitOnInput = batch || !checkpointingEnabled || hasUpstreamCommitter(inputStream);
+
+        // Create a global shuffle and add the global committer with parallelism 1.
+        final PhysicalTransformation<Void> transformation =
+                (PhysicalTransformation<Void>)
+                        inputStream
+                                .global()
+                                .transform(
+                                        GLOBAL_COMMITTER_TRANSFORMATION_NAME,
+                                        Types.VOID,
+                                        new GlobalCommitterOperator<>(
+                                                globalCommitterTransform.getCommitterFactory(),
+                                                globalCommitterTransform.getCommittableSerializer(),
+                                                commitOnInput))
+                                .getTransformation();
+        transformation.setChainingStrategy(ChainingStrategy.ALWAYS);
+        transformation.setName(GLOBAL_COMMITTER_TRANSFORMATION_NAME);
+        transformation.setParallelism(1);
+        transformation.setMaxParallelism(1);
+        return Collections.emptyList();
+    }
+
+    /**
+     * Looks for a committer in the pipeline and aborts on writer. The GlobalCommitter behaves
+     * differently if there is a committer after the writer.
+     */
+    private static boolean hasUpstreamCommitter(DataStream<?> ds) {
+        Transformation<?> dsTransformation = ds.getTransformation();
+
+        Set<Integer> seenIds = new HashSet<>();
+        Queue<Transformation<?>> pendingsTransformations =
+                new ArrayDeque<>(Collections.singleton(dsTransformation));
+        while (!pendingsTransformations.isEmpty()) {
+            Transformation<?> transformation = pendingsTransformations.poll();
+            if (transformation instanceof OneInputTransformation) {
+                StreamOperatorFactory<?> operatorFactory =
+                        ((OneInputTransformation<?, ?>) transformation).getOperatorFactory();
+                if (operatorFactory instanceof CommitterOperatorFactory) {
+                    // found the committer, so the global committer should commit on input
+                    return true;
+                }
+                if (operatorFactory instanceof SinkWriterOperatorFactory) {
+                    // don't look at the inputs of the writer
+                    continue;
+                }
+            }
+            for (Transformation<?> input : transformation.getInputs()) {
+                if (seenIds.add(input.getId())) {
+                    pendingsTransformations.add(input);
+                }
+            }
+        }
+
+        return false;
+    }
+}
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/GlobalCommitterOperatorTest.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/GlobalCommitterOperatorTest.java
index dc45e939b1c..865a24175ae 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/GlobalCommitterOperatorTest.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/GlobalCommitterOperatorTest.java
@@ -24,6 +24,8 @@ import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
 
 import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
 
 import java.io.IOException;
 import java.util.ArrayList;
@@ -35,66 +37,112 @@ import static org.assertj.core.api.Assertions.assertThat;
 
 class GlobalCommitterOperatorTest {
 
-    @Test
-    void testWaitForCommittablesOfLatestCheckpointBeforeCommitting() throws Exception {
+    @ParameterizedTest
+    @ValueSource(booleans = {true, false})
+    void testWaitForCommittablesOfLatestCheckpointBeforeCommitting(boolean commitOnInput)
+            throws Exception {
         final MockCommitter committer = new MockCommitter();
-        final OneInputStreamOperatorTestHarness<CommittableMessage<Integer>, Void> testHarness =
-                createTestHarness(committer);
-        testHarness.open();
+        try (OneInputStreamOperatorTestHarness<CommittableMessage<Integer>, Void> testHarness =
+                createTestHarness(committer, commitOnInput)) {
+            testHarness.open();
 
-        testHarness.processElement(new StreamRecord<>(new CommittableSummary<>(1, 1, 1L, 2, 0, 0)));
+            long cid = 1L;
+            testHarness.processElement(
+                    new StreamRecord<>(new CommittableSummary<>(1, 1, cid, 2, 0, 0)));
 
-        testHarness.processElement(new StreamRecord<>(new CommittableWithLineage<>(1, 1L, 1)));
+            testHarness.processElement(new StreamRecord<>(new CommittableWithLineage<>(1, cid, 1)));
 
-        testHarness.notifyOfCompletedCheckpoint(1);
+            testHarness.notifyOfCompletedCheckpoint(cid);
 
-        assertThat(testHarness.getOutput()).isEmpty();
-        // Not committed because incomplete
-        assertThat(committer.committed).isEmpty();
+            assertThat(testHarness.getOutput()).isEmpty();
+            // Not committed because incomplete
+            assertThat(committer.committed).isEmpty();
 
-        testHarness.processElement(new StreamRecord<>(new CommittableWithLineage<>(2, 1L, 1)));
+            // immediately commit on receiving the second committable iff commitOnInput is true
+            testHarness.processElement(new StreamRecord<>(new CommittableWithLineage<>(2, cid, 1)));
+            if (commitOnInput) {
+                assertThat(committer.committed).containsExactly(1, 2);
+            } else {
+                assertThat(committer.committed).isEmpty();
+                testHarness.notifyOfCompletedCheckpoint(cid + 1);
+                assertThat(committer.committed).containsExactly(1, 2);
+            }
 
-        testHarness.notifyOfCompletedCheckpoint(2);
-
-        assertThat(testHarness.getOutput()).isEmpty();
-        assertThat(committer.committed).containsExactly(1, 2);
-        testHarness.close();
+            assertThat(testHarness.getOutput()).isEmpty();
+        }
     }
 
-    @Test
-    void testStateRestore() throws Exception {
+    @ParameterizedTest
+    @ValueSource(booleans = {true, false})
+    void testWaitForNotifyCheckpointCompleted(boolean commitOnInput) throws Exception {
         final MockCommitter committer = new MockCommitter();
-        final OneInputStreamOperatorTestHarness<CommittableMessage<Integer>, Void> testHarness =
-                createTestHarness(committer);
-        testHarness.open();
+        try (OneInputStreamOperatorTestHarness<CommittableMessage<Integer>, Void> testHarness =
+                createTestHarness(committer, commitOnInput)) {
+            testHarness.open();
 
-        final CommittableSummary<Integer> committableSummary =
-                new CommittableSummary<>(1, 1, 0L, 1, 1, 0);
-        testHarness.processElement(new StreamRecord<>(committableSummary));
-        final CommittableWithLineage<Integer> first = new CommittableWithLineage<>(1, 0L, 1);
-        testHarness.processElement(new StreamRecord<>(first));
+            long cid = 1L;
+            testHarness.processElement(
+                    new StreamRecord<>(new CommittableSummary<>(1, 1, cid, 2, 0, 0)));
 
-        final OperatorSubtaskState snapshot = testHarness.snapshot(0L, 2L);
-        assertThat(testHarness.getOutput()).isEmpty();
-        testHarness.close();
-        assertThat(committer.committed).isEmpty();
+            testHarness.processElement(new StreamRecord<>(new CommittableWithLineage<>(1, cid, 1)));
 
-        final OneInputStreamOperatorTestHarness<CommittableMessage<Integer>, Void> restored =
-                createTestHarness(committer);
+            assertThat(testHarness.getOutput()).isEmpty();
+            // Not committed because incomplete
+            assertThat(committer.committed).isEmpty();
 
-        restored.initializeState(snapshot);
-        restored.open();
+            // immediately commit on receiving the second committable iff commitOnInput is true
+            testHarness.processElement(new StreamRecord<>(new CommittableWithLineage<>(2, cid, 1)));
+            if (commitOnInput) {
+                assertThat(committer.committed).containsExactly(1, 2);
+            } else {
+                assertThat(committer.committed).isEmpty();
+            }
 
-        assertThat(testHarness.getOutput()).isEmpty();
-        assertThat(committer.committed).containsExactly(1);
-        restored.close();
+            // for commitOnInput = false, the committer waits for notifyCheckpointComplete
+            testHarness.notifyOfCompletedCheckpoint(cid);
+
+            assertThat(committer.committed).containsExactly(1, 2);
+
+            assertThat(testHarness.getOutput()).isEmpty();
+        }
     }
 
     @Test
-    void testCommitAllCommittablesOnEndOfInput() throws Exception {
+    void testStateRestore() throws Exception {
+        final MockCommitter committer = new MockCommitter();
+        try (OneInputStreamOperatorTestHarness<CommittableMessage<Integer>, Void> testHarness =
+                createTestHarness(committer, false)) {
+            testHarness.open();
+
+            final CommittableSummary<Integer> committableSummary =
+                    new CommittableSummary<>(1, 1, 0L, 1, 1, 0);
+            testHarness.processElement(new StreamRecord<>(committableSummary));
+            final CommittableWithLineage<Integer> first = new CommittableWithLineage<>(1, 0L, 1);
+            testHarness.processElement(new StreamRecord<>(first));
+
+            final OperatorSubtaskState snapshot = testHarness.snapshot(0L, 2L);
+            assertThat(testHarness.getOutput()).isEmpty();
+            testHarness.close();
+            assertThat(committer.committed).isEmpty();
+
+            try (OneInputStreamOperatorTestHarness<CommittableMessage<Integer>, Void> restored =
+                    createTestHarness(committer, true)) {
+
+                restored.initializeState(snapshot);
+                restored.open();
+
+                assertThat(testHarness.getOutput()).isEmpty();
+                assertThat(committer.committed).containsExactly(1);
+            }
+        }
+    }
+
+    @ParameterizedTest
+    @ValueSource(booleans = {true, false})
+    void testCommitAllCommittablesOnFinalCheckpoint(boolean commitOnInput) throws Exception {
         final MockCommitter committer = new MockCommitter();
         final OneInputStreamOperatorTestHarness<CommittableMessage<Integer>, Void> testHarness =
-                createTestHarness(committer);
+                createTestHarness(committer, commitOnInput);
         testHarness.open();
 
         final CommittableSummary<Integer> committableSummary =
@@ -109,16 +157,23 @@ class GlobalCommitterOperatorTest {
         final CommittableWithLineage<Integer> second = new CommittableWithLineage<>(2, EOI, 2);
         testHarness.processElement(new StreamRecord<>(second));
 
-        testHarness.endInput();
+        // commitOnInput implies that the global committer is not using notifyCheckpointComplete
+        if (commitOnInput) {
+            assertThat(committer.committed).containsExactly(1, 2);
+        } else {
+            assertThat(committer.committed).isEmpty();
+            testHarness.notifyOfCompletedCheckpoint(EOI);
+            assertThat(committer.committed).containsExactly(1, 2);
+        }
 
         assertThat(testHarness.getOutput()).isEmpty();
-        assertThat(committer.committed).containsExactly(1, 2);
     }
 
     private OneInputStreamOperatorTestHarness<CommittableMessage<Integer>, Void> createTestHarness(
-            Committer<Integer> committer) throws Exception {
+            Committer<Integer> committer, boolean commitOnInput) throws Exception {
         return new OneInputStreamOperatorTestHarness<>(
-                new GlobalCommitterOperator<>(() -> committer, IntegerSerializer::new));
+                new GlobalCommitterOperator<>(
+                        () -> committer, IntegerSerializer::new, commitOnInput));
     }
 
     private static class MockCommitter implements Committer<Integer> {
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollectorTest.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollectorTest.java
index b33faafa8dd..b8a58aa6c65 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollectorTest.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollectorTest.java
@@ -25,6 +25,8 @@ import org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions;
 
 import org.junit.jupiter.api.Test;
 
+import java.util.Optional;
+
 import static org.apache.flink.streaming.api.connector.sink2.CommittableMessage.EOI;
 import static org.assertj.core.api.Assertions.assertThat;
 
@@ -44,7 +46,7 @@ class CommittableCollectorTest {
 
         assertThat(committableCollector.getCheckpointCommittablesUpTo(2)).hasSize(2);
 
-        assertThat(committableCollector.getEndOfInputCommittable()).isNull();
+        assertThat(committableCollector.getEndOfInputCommittable()).isNotPresent();
     }
 
     @Test
@@ -54,9 +56,10 @@ class CommittableCollectorTest {
         CommittableSummary<Integer> first = new CommittableSummary<>(1, 1, EOI, 1, 0, 0);
         committableCollector.addMessage(first);
 
-        CheckpointCommittableManager<Integer> endOfInputCommittable =
+        Optional<CheckpointCommittableManager<Integer>> endOfInputCommittable =
                 committableCollector.getEndOfInputCommittable();
-        assertThat(endOfInputCommittable).isNotNull();
-        SinkV2Assertions.assertThat(endOfInputCommittable.getSummary(1, 1)).hasCheckpointId(EOI);
+        assertThat(endOfInputCommittable).isPresent();
+        SinkV2Assertions.assertThat(endOfInputCommittable.get().getSummary(1, 1))
+                .hasCheckpointId(EOI);
     }
 }


(flink) 03/04: [FLINK-36379] Optimize committers with UC disabled

Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

arvid pushed a commit to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git

commit 67932dc1f4927726c6ce541d7a4ae84359f571b0
Author: Arvid Heise <AH...@users.noreply.github.com>
AuthorDate: Fri Oct 4 11:15:24 2024 +0200

    [FLINK-36379] Optimize committers with UC disabled
    
    Without UCs, a committer doesn't need to do anything on #processInput except collecting. It emits only on notifyCheckpointCompleted (or endInput for batch).
    
    We can also harden some contracts:
    * NotifyCheckpointCompleted can assert that all committables are received.
    * Emit committables downstream only if all committables are finished.
    
    (cherry picked from commit 7f40ab97258bb323d11ef3c7efcf8761fd34bce0)
---
 .../runtime/operators/sink/CommitterOperator.java  |  9 +------
 .../CheckpointCommittableManagerImpl.java          | 29 ++++++++++++++++++++--
 .../operators/sink/CommitterOperatorTestBase.java  |  3 ++-
 .../CheckpointCommittableManagerImplTest.java      | 13 +++++-----
 4 files changed, 37 insertions(+), 17 deletions(-)

diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java
index 02db28fa8a3..d155f5dd509 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperator.java
@@ -181,7 +181,7 @@ class CommitterOperator<CommT> extends AbstractStreamOperator<CommittableMessage
     private void commitAndEmit(CheckpointCommittableManager<CommT> committableManager)
             throws IOException, InterruptedException {
         Collection<CommittableWithLineage<CommT>> committed = committableManager.commit(committer);
-        if (emitDownstream && !committed.isEmpty()) {
+        if (emitDownstream && committableManager.isFinished()) {
             int subtaskId = getRuntimeContext().getTaskInfo().getIndexOfThisSubtask();
             int numberOfSubtasks = getRuntimeContext().getTaskInfo().getNumberOfParallelSubtasks();
             output.collect(
@@ -201,13 +201,6 @@ class CommitterOperator<CommT> extends AbstractStreamOperator<CommittableMessage
     @Override
     public void processElement(StreamRecord<CommittableMessage<CommT>> element) throws Exception {
         committableCollector.addMessage(element.getValue());
-
-        // in case of unaligned checkpoint, we may receive notifyCheckpointComplete before the
-        // committables
-        long checkpointId = element.getValue().getCheckpointIdOrEOI();
-        if (checkpointId <= lastCompletedCheckpointId) {
-            commitAndEmitCheckpoints();
-        }
     }
 
     @Override
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java
index 99d321d4cd1..bb6cceead47 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java
@@ -24,6 +24,7 @@ import org.apache.flink.metrics.groups.SinkCommitterMetricGroup;
 import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
 import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
 import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.util.Preconditions;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -168,13 +169,37 @@ class CheckpointCommittableManagerImpl<CommT> implements CheckpointCommittableMa
         return committed;
     }
 
-    Collection<CommitRequestImpl<CommT>> getPendingRequests(boolean onlyIfFullyReceived) {
+    Collection<CommitRequestImpl<CommT>> getPendingRequests(boolean assertFull) {
         return subtasksCommittableManagers.values().stream()
-                .filter(subtask -> !onlyIfFullyReceived || subtask.hasReceivedAll())
+                .peek(subtask -> assertReceivedAll(assertFull, subtask))
                 .flatMap(SubtaskCommittableManager::getPendingRequests)
                 .collect(Collectors.toList());
     }
 
+    /**
+     * For committers: Sinks don't use unaligned checkpoints, so we receive all committables of a
+     * given upstream task before the respective barrier. Thus, when the barrier reaches the
+     * committer, all committables of a specific checkpoint must have been received. Committing
+     * happens even later on notifyCheckpointComplete.
+     *
+     * <p>Global committers need to ensure that all committables of all subtasks have been received
+     * with {@link #hasGloballyReceivedAll()} before trying to commit. Naturally, this method then
+     * becomes a no-op.
+     *
+     * <p>Note that by transitivity, the assertion also holds for committables of subsumed
+     * checkpoints.
+     *
+     * <p>This assertion will fail in case of bugs in the writer or in the pre-commit topology if
+     * present.
+     */
+    private void assertReceivedAll(boolean assertFull, SubtaskCommittableManager<CommT> subtask) {
+        Preconditions.checkArgument(
+                !assertFull || subtask.hasReceivedAll(),
+                "Trying to commit incomplete batch of committables subtask=%s, manager=%s",
+                subtask.getSubtaskId(),
+                this);
+    }
+
     Collection<CommittableWithLineage<CommT>> drainFinished() {
         return subtasksCommittableManagers.values().stream()
                 .flatMap(subtask -> subtask.drainCommitted().stream())
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperatorTestBase.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperatorTestBase.java
index c24721884f6..e63ec66c41b 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperatorTestBase.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperatorTestBase.java
@@ -101,7 +101,8 @@ abstract class CommitterOperatorTestBase {
         final CommittableWithLineage<String> first = new CommittableWithLineage<>("1", 1L, 1);
         testHarness.processElement(new StreamRecord<>(first));
 
-        testHarness.notifyOfCompletedCheckpoint(1);
+        assertThatCode(() -> testHarness.notifyOfCompletedCheckpoint(1))
+                .hasMessageContaining("Trying to commit incomplete batch of committables");
 
         assertThat(testHarness.getOutput()).isEmpty();
         assertThat(sinkAndCounters.commitCounter.getAsInt()).isZero();
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImplTest.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImplTest.java
index b9c65c7c167..fcc2f559043 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImplTest.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImplTest.java
@@ -34,6 +34,7 @@ import java.util.Collection;
 import java.util.HashMap;
 
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatCode;
 import static org.assertj.core.api.Assertions.assertThatThrownBy;
 
 class CheckpointCommittableManagerImplTest {
@@ -73,20 +74,20 @@ class CheckpointCommittableManagerImplTest {
 
         final Committer<Integer> committer = new NoOpCommitter();
         // Only commit fully received committables
-        assertThat(checkpointCommittables.commit(committer))
-                .hasSize(1)
-                .satisfiesExactly(c -> assertThat(c.getCommittable()).isEqualTo(3));
+        assertThatCode(() -> checkpointCommittables.commit(committer))
+                .hasMessageContaining("Trying to commit incomplete batch of committables");
 
         // Even on retry
-        assertThat(checkpointCommittables.commit(committer)).isEmpty();
+        assertThatCode(() -> checkpointCommittables.commit(committer))
+                .hasMessageContaining("Trying to commit incomplete batch of committables");
 
         // Add missing committable
         checkpointCommittables.addCommittable(new CommittableWithLineage<>(5, 1L, 2));
         // Commit all committables
         assertThat(checkpointCommittables.commit(committer))
-                .hasSize(2)
+                .hasSize(3)
                 .extracting(CommittableWithLineage::getCommittable)
-                .containsExactlyInAnyOrder(4, 5);
+                .containsExactlyInAnyOrder(3, 4, 5);
     }
 
     @Test


(flink) 01/04: [FLINK-36379] Refactor sink test assertions

Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

arvid pushed a commit to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git

commit a613d2c2466b4e0ec7c20d77274aeb3e39e51079
Author: Arvid Heise <ar...@apache.org>
AuthorDate: Wed Oct 2 09:55:16 2024 +0200

    [FLINK-36379] Refactor sink test assertions
    
    Use more of the assertj native patterns to compare results.
---
 .../file/sink/compactor/CompactorOperatorTest.java | 107 ++++++-------
 .../connector/sink2/CommittableSummaryAssert.java  |  18 +--
 ...sert.java => CommittableWithLineageAssert.java} |  15 +-
 .../api/connector/sink2/SinkV2Assertions.java      |  28 +++-
 .../operators/sink/CommitterOperatorTestBase.java  |  72 +++++----
 .../runtime/operators/sink/SinkTestUtil.java       |  64 --------
 .../operators/sink/SinkWriterOperatorTestBase.java | 166 ++++++++-------------
 .../CommittableCollectorSerializerTest.java        |  64 +++-----
 .../SubtaskCommittableManagerTest.java             |  22 ++-
 9 files changed, 221 insertions(+), 335 deletions(-)

diff --git a/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/sink/compactor/CompactorOperatorTest.java b/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/sink/compactor/CompactorOperatorTest.java
index 51be721418e..d5b50e26cfd 100644
--- a/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/sink/compactor/CompactorOperatorTest.java
+++ b/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/sink/compactor/CompactorOperatorTest.java
@@ -33,7 +33,6 @@ import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
 import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
 import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
 import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
-import org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions;
 import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
 import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
 import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter;
@@ -45,6 +44,7 @@ import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
 import org.apache.flink.types.Either;
 
+import org.assertj.core.api.ListAssert;
 import org.junit.jupiter.api.Test;
 
 import java.io.BufferedWriter;
@@ -57,6 +57,9 @@ import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
 
+import static org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions.committableSummary;
+import static org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions.committableWithLineage;
+import static org.assertj.core.api.Assertions.as;
 import static org.assertj.core.api.Assertions.assertThat;
 
 /** Test for {@link CompactorOperator}. */
@@ -93,16 +96,13 @@ class CompactorOperatorTest extends AbstractCompactTestBase {
             harness.prepareSnapshotPreBarrier(2);
 
             // 1summary+1compacted+2cleanup
-            List<CommittableMessage<FileSinkCommittable>> results = harness.extractOutputValues();
-            assertThat(results).hasSize(4);
-            SinkV2Assertions.assertThat((CommittableSummary<?>) results.get(0))
-                    .hasPendingCommittables(3);
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(1))
+            ListAssert<CommittableMessage<FileSinkCommittable>> results =
+                    assertThat(harness.extractOutputValues()).hasSize(4);
+            results.element(0, as(committableSummary())).hasPendingCommittables(3);
+            results.element(1, as(committableWithLineage()))
                     .hasCommittable(committable("0", "compacted-0", 10));
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(2))
-                    .hasCommittable(cleanupPath("0", ".0"));
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(3))
-                    .hasCommittable(cleanupPath("0", ".1"));
+            results.element(2, as(committableWithLineage())).hasCommittable(cleanupPath("0", ".0"));
+            results.element(3, as(committableWithLineage())).hasCommittable(cleanupPath("0", ".1"));
         }
     }
 
@@ -138,14 +138,12 @@ class CompactorOperatorTest extends AbstractCompactTestBase {
 
             harness.prepareSnapshotPreBarrier(2);
 
-            List<CommittableMessage<FileSinkCommittable>> results = harness.extractOutputValues();
-            assertThat(results).hasSize(3);
-            SinkV2Assertions.assertThat((CommittableSummary<?>) results.get(0))
-                    .hasPendingCommittables(2);
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(1))
+            ListAssert<CommittableMessage<FileSinkCommittable>> messages =
+                    assertThat(harness.extractOutputValues()).hasSize(3);
+            messages.element(0, as(committableSummary())).hasPendingCommittables(2);
+            messages.element(1, as(committableWithLineage()))
                     .hasCommittable(cleanupInprogressRequest);
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(2))
-                    .hasCommittable(cleanupPathRequest);
+            messages.element(2, as(committableWithLineage())).hasCommittable(cleanupPathRequest);
         }
     }
 
@@ -207,25 +205,19 @@ class CompactorOperatorTest extends AbstractCompactTestBase {
             assertThat(harness.extractOutputValues()).hasSize(8);
 
             // 1summary+1compacted+2cleanup * 2
-            List<CommittableMessage<FileSinkCommittable>> results = harness.extractOutputValues();
-            assertThat(results).hasSize(8);
-            SinkV2Assertions.assertThat((CommittableSummary<?>) results.get(0))
-                    .hasPendingCommittables(3);
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(1))
+            ListAssert<CommittableMessage<FileSinkCommittable>> results =
+                    assertThat(harness.extractOutputValues()).hasSize(8);
+            results.element(0, as(committableSummary())).hasPendingCommittables(3);
+            results.element(1, as(committableWithLineage()))
                     .hasCommittable(committable("0", "compacted-0", 10));
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(2))
-                    .hasCommittable(cleanupPath("0", ".0"));
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(3))
-                    .hasCommittable(cleanupPath("0", ".1"));
-
-            SinkV2Assertions.assertThat((CommittableSummary<?>) results.get(4))
-                    .hasPendingCommittables(3);
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(5))
+            results.element(2, as(committableWithLineage())).hasCommittable(cleanupPath("0", ".0"));
+            results.element(3, as(committableWithLineage())).hasCommittable(cleanupPath("0", ".1"));
+
+            results.element(4, as(committableSummary())).hasPendingCommittables(3);
+            results.element(5, as(committableWithLineage()))
                     .hasCommittable(committable("0", "compacted-2", 10));
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(6))
-                    .hasCommittable(cleanupPath("0", ".2"));
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(7))
-                    .hasCommittable(cleanupPath("0", ".3"));
+            results.element(6, as(committableWithLineage())).hasCommittable(cleanupPath("0", ".2"));
+            results.element(7, as(committableWithLineage())).hasCommittable(cleanupPath("0", ".3"));
         }
     }
 
@@ -317,10 +309,9 @@ class CompactorOperatorTest extends AbstractCompactTestBase {
 
             // 1 summary + (1 compacted committable + 1 compacted cleanup) * 6 + 1 hidden + 1 normal
             // + 1 summary + 1 cleanup + 1 summary
-            List<CommittableMessage<FileSinkCommittable>> results = harness.extractOutputValues();
-            assertThat(results).hasSize(18);
-            SinkV2Assertions.assertThat((CommittableSummary<?>) results.get(0))
-                    .hasPendingCommittables(14);
+            ListAssert<CommittableMessage<FileSinkCommittable>> results =
+                    assertThat(harness.extractOutputValues()).hasSize(18);
+            results.element(0, as(committableSummary())).hasPendingCommittables(14);
 
             List<FileSinkCommittable> expectedResult =
                     Arrays.asList(
@@ -340,17 +331,15 @@ class CompactorOperatorTest extends AbstractCompactTestBase {
                             committable("0", "7", 8));
 
             for (int i = 0; i < expectedResult.size(); ++i) {
-                SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(i + 1))
+                results.element(i + 1, as(committableWithLineage()))
                         .hasCommittable(expectedResult.get(i));
             }
 
-            SinkV2Assertions.assertThat((CommittableSummary<?>) results.get(15))
-                    .hasPendingCommittables(1);
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(16))
+            results.element(15, as(committableSummary())).hasPendingCommittables(1);
+            results.element(16, as(committableWithLineage()))
                     .hasCommittable(cleanupPath("0", ".6"));
 
-            SinkV2Assertions.assertThat((CommittableSummary<?>) results.get(17))
-                    .hasPendingCommittables(3);
+            results.element(17, as(committableSummary())).hasPendingCommittables(3);
         }
     }
 
@@ -386,14 +375,12 @@ class CompactorOperatorTest extends AbstractCompactTestBase {
 
             state = harness.snapshot(1, 1L);
 
-            List<CommittableMessage<FileSinkCommittable>> results = harness.extractOutputValues();
-            assertThat(results).hasSize(3);
-            SinkV2Assertions.assertThat((CommittableSummary<?>) results.get(0))
-                    .hasPendingCommittables(4);
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(1))
+            ListAssert<CommittableMessage<FileSinkCommittable>> results =
+                    assertThat(harness.extractOutputValues()).hasSize(3);
+            results.element(0, as(committableSummary())).hasPendingCommittables(4);
+            results.element(1, as(committableWithLineage()))
                     .hasCommittable(committable("0", "compacted-1", 1));
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(2))
-                    .hasCommittable(cleanupPath("0", ".1"));
+            results.element(2, as(committableWithLineage())).hasCommittable(cleanupPath("0", ".1"));
         }
 
         try (OneInputStreamOperatorTestHarness<
@@ -422,11 +409,11 @@ class CompactorOperatorTest extends AbstractCompactTestBase {
 
             state = harness.snapshot(2, 2L);
 
-            List<CommittableMessage<FileSinkCommittable>> results = harness.extractOutputValues();
-            assertThat(results).hasSize(2);
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(0))
+            ListAssert<CommittableMessage<FileSinkCommittable>> results =
+                    assertThat(harness.extractOutputValues()).hasSize(2);
+            results.element(0, as(committableWithLineage()))
                     .hasCommittable(committable("0", "2", 2));
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(1))
+            results.element(1, as(committableWithLineage()))
                     .hasCommittable(committable("0", "3", 3));
         }
 
@@ -445,12 +432,10 @@ class CompactorOperatorTest extends AbstractCompactTestBase {
             harness.processElement(
                     new StreamRecord<>(Either.Left(new CommittableSummary<>(0, 1, 2L, 0, 0, 0))));
 
-            List<CommittableMessage<FileSinkCommittable>> results = harness.extractOutputValues();
-            assertThat(results).hasSize(2);
-            SinkV2Assertions.assertThat((CommittableSummary<?>) results.get(0))
-                    .hasPendingCommittables(1);
-            SinkV2Assertions.assertThat((CommittableWithLineage<?>) results.get(1))
-                    .hasCommittable(cleanupPath("0", ".2"));
+            ListAssert<CommittableMessage<FileSinkCommittable>> results =
+                    assertThat(harness.extractOutputValues()).hasSize(2);
+            results.element(0, as(committableSummary())).hasPendingCommittables(1);
+            results.element(1, as(committableWithLineage())).hasCommittable(cleanupPath("0", ".2"));
         }
     }
 
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/CommittableSummaryAssert.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/CommittableSummaryAssert.java
index ed6ea6440ee..14416da2d99 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/CommittableSummaryAssert.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/CommittableSummaryAssert.java
@@ -21,34 +21,34 @@ package org.apache.flink.streaming.api.connector.sink2;
 import org.assertj.core.api.AbstractObjectAssert;
 
 /** Custom assertions for {@link CommittableSummary}. */
-public class CommittableSummaryAssert
-        extends AbstractObjectAssert<CommittableSummaryAssert, CommittableSummary<?>> {
+public class CommittableSummaryAssert<CommT>
+        extends AbstractObjectAssert<CommittableSummaryAssert<CommT>, CommittableSummary<CommT>> {
 
-    public CommittableSummaryAssert(CommittableSummary<?> summary) {
+    public CommittableSummaryAssert(CommittableSummary<CommT> summary) {
         super(summary, CommittableSummaryAssert.class);
     }
 
-    public CommittableSummaryAssert hasSubtaskId(int subtaskId) {
+    public CommittableSummaryAssert<CommT> hasSubtaskId(int subtaskId) {
         return returns(subtaskId, CommittableSummary::getSubtaskId);
     }
 
-    public CommittableSummaryAssert hasNumberOfSubtasks(int numberOfSubtasks) {
+    public CommittableSummaryAssert<CommT> hasNumberOfSubtasks(int numberOfSubtasks) {
         return returns(numberOfSubtasks, CommittableSummary::getNumberOfSubtasks);
     }
 
-    public CommittableSummaryAssert hasOverallCommittables(int committableNumber) {
+    public CommittableSummaryAssert<CommT> hasOverallCommittables(int committableNumber) {
         return returns(committableNumber, CommittableSummary::getNumberOfCommittables);
     }
 
-    public CommittableSummaryAssert hasPendingCommittables(int committableNumber) {
+    public CommittableSummaryAssert<CommT> hasPendingCommittables(int committableNumber) {
         return returns(committableNumber, CommittableSummary::getNumberOfPendingCommittables);
     }
 
-    public CommittableSummaryAssert hasFailedCommittables(int committableNumber) {
+    public CommittableSummaryAssert<CommT> hasFailedCommittables(int committableNumber) {
         return returns(committableNumber, CommittableSummary::getNumberOfFailedCommittables);
     }
 
-    public CommittableSummaryAssert hasCheckpointId(long checkpointId) {
+    public CommittableSummaryAssert<CommT> hasCheckpointId(long checkpointId) {
         return returns(checkpointId, CommittableSummary::getCheckpointIdOrEOI);
     }
 }
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/CommittableWithLinageAssert.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/CommittableWithLineageAssert.java
similarity index 70%
rename from flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/CommittableWithLinageAssert.java
rename to flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/CommittableWithLineageAssert.java
index 853fe6235d7..1e0dcc1a811 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/CommittableWithLinageAssert.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/CommittableWithLineageAssert.java
@@ -24,22 +24,23 @@ import org.assertj.core.api.AbstractObjectAssert;
  * Custom assertions for {@link
  * org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage}.
  */
-public class CommittableWithLinageAssert
-        extends AbstractObjectAssert<CommittableWithLinageAssert, CommittableWithLineage<?>> {
+public class CommittableWithLineageAssert<CommT>
+        extends AbstractObjectAssert<
+                CommittableWithLineageAssert<CommT>, CommittableWithLineage<?>> {
 
-    public CommittableWithLinageAssert(CommittableWithLineage<?> summary) {
-        super(summary, CommittableWithLinageAssert.class);
+    public CommittableWithLineageAssert(CommittableWithLineage<?> summary) {
+        super(summary, CommittableWithLineageAssert.class);
     }
 
-    public CommittableWithLinageAssert hasCommittable(Object committable) {
+    public CommittableWithLineageAssert<CommT> hasCommittable(CommT committable) {
         return returns(committable, CommittableWithLineage::getCommittable);
     }
 
-    public CommittableWithLinageAssert hasCheckpointId(long checkpointId) {
+    public CommittableWithLineageAssert<CommT> hasCheckpointId(long checkpointId) {
         return returns(checkpointId, CommittableWithLineage::getCheckpointIdOrEOI);
     }
 
-    public CommittableWithLinageAssert hasSubtaskId(int subtaskId) {
+    public CommittableWithLineageAssert<CommT> hasSubtaskId(int subtaskId) {
         return returns(subtaskId, CommittableWithLineage::getSubtaskId);
     }
 }
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/SinkV2Assertions.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/SinkV2Assertions.java
index 1ea890024ff..2e810550319 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/SinkV2Assertions.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/connector/sink2/SinkV2Assertions.java
@@ -18,15 +18,33 @@
 
 package org.apache.flink.streaming.api.connector.sink2;
 
+import org.assertj.core.api.InstanceOfAssertFactory;
+
 /** Custom assertions for Sink V2 related classes. */
 public class SinkV2Assertions {
+    @SuppressWarnings({"rawtypes"})
+    public static <CommT>
+            InstanceOfAssertFactory<CommittableWithLineage, CommittableWithLineageAssert<CommT>>
+                    committableWithLineage() {
+        return new InstanceOfAssertFactory<>(
+                CommittableWithLineage.class, SinkV2Assertions::<CommT>assertThat);
+    }
+
+    @SuppressWarnings({"rawtypes"})
+    public static <CommT>
+            InstanceOfAssertFactory<CommittableSummary, CommittableSummaryAssert<CommT>>
+                    committableSummary() {
+        return new InstanceOfAssertFactory<>(
+                CommittableSummary.class, SinkV2Assertions::<CommT>assertThat);
+    }
 
-    public static CommittableSummaryAssert assertThat(CommittableSummary<?> summary) {
-        return new CommittableSummaryAssert(summary);
+    public static <CommT> CommittableSummaryAssert<CommT> assertThat(
+            CommittableSummary<CommT> summary) {
+        return new CommittableSummaryAssert<>(summary);
     }
 
-    public static CommittableWithLinageAssert assertThat(
-            CommittableWithLineage<?> committableWithLineage) {
-        return new CommittableWithLinageAssert(committableWithLineage);
+    public static <CommT> CommittableWithLineageAssert<CommT> assertThat(
+            CommittableWithLineage<CommT> committableWithLineage) {
+        return new CommittableWithLineageAssert<>(committableWithLineage);
     }
 }
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperatorTestBase.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperatorTestBase.java
index 096e2202dcb..c24721884f6 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperatorTestBase.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/CommitterOperatorTestBase.java
@@ -23,22 +23,20 @@ import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
 import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
 import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
 import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
-import org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions;
-import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
 
+import org.assertj.core.api.ListAssert;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.params.ParameterizedTest;
 import org.junit.jupiter.params.provider.ValueSource;
 
-import java.util.List;
 import java.util.function.IntSupplier;
 
 import static org.apache.flink.streaming.api.connector.sink2.CommittableMessage.EOI;
-import static org.apache.flink.streaming.runtime.operators.sink.SinkTestUtil.fromOutput;
-import static org.apache.flink.streaming.runtime.operators.sink.SinkTestUtil.toCommittableSummary;
-import static org.apache.flink.streaming.runtime.operators.sink.SinkTestUtil.toCommittableWithLinage;
+import static org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions.committableSummary;
+import static org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions.committableWithLineage;
+import static org.assertj.core.api.Assertions.as;
 import static org.assertj.core.api.Assertions.assertThat;
 import static org.assertj.core.api.Assertions.assertThatCode;
 
@@ -73,12 +71,13 @@ abstract class CommitterOperatorTestBase {
 
         assertThat(sinkAndCounters.commitCounter.getAsInt()).isEqualTo(1);
         if (withPostCommitTopology) {
-            final List<StreamElement> output = fromOutput(testHarness.getOutput());
-            SinkV2Assertions.assertThat(toCommittableSummary(output.get(0)))
+            ListAssert<CommittableMessage<String>> records =
+                    assertThat(testHarness.extractOutputValues()).hasSize(2);
+            records.element(0, as(committableSummary()))
                     .hasFailedCommittables(committableSummary.getNumberOfFailedCommittables())
                     .hasOverallCommittables(committableSummary.getNumberOfCommittables())
                     .hasPendingCommittables(0);
-            SinkV2Assertions.assertThat(toCommittableWithLinage(output.get(1)))
+            records.element(1, as(committableWithLineage()))
                     .isEqualTo(committableWithLineage.withSubtaskId(0));
         } else {
             assertThat(testHarness.getOutput()).isEmpty();
@@ -112,17 +111,15 @@ abstract class CommitterOperatorTestBase {
 
         assertThatCode(() -> testHarness.notifyOfCompletedCheckpoint(1)).doesNotThrowAnyException();
 
-        final List<StreamElement> output = fromOutput(testHarness.getOutput());
-        assertThat(output).hasSize(3);
         assertThat(sinkAndCounters.commitCounter.getAsInt()).isEqualTo(2);
-        SinkV2Assertions.assertThat(toCommittableSummary(output.get(0)))
+        ListAssert<CommittableMessage<String>> records =
+                assertThat(testHarness.extractOutputValues()).hasSize(3);
+        records.element(0, as(committableSummary()))
                 .hasFailedCommittables(committableSummary.getNumberOfFailedCommittables())
                 .hasOverallCommittables(committableSummary.getNumberOfCommittables())
                 .hasPendingCommittables(0);
-        SinkV2Assertions.assertThat(toCommittableWithLinage(output.get(1)))
-                .isEqualTo(first.withSubtaskId(0));
-        SinkV2Assertions.assertThat(toCommittableWithLinage(output.get(2)))
-                .isEqualTo(second.withSubtaskId(0));
+        records.element(1, as(committableWithLineage())).isEqualTo(first.withSubtaskId(0));
+        records.element(2, as(committableWithLineage())).isEqualTo(second.withSubtaskId(0));
         testHarness.close();
     }
 
@@ -154,16 +151,14 @@ abstract class CommitterOperatorTestBase {
             testHarness.notifyOfCompletedCheckpoint(1);
         }
 
-        final List<StreamElement> output = fromOutput(testHarness.getOutput());
-        assertThat(output).hasSize(3);
-        SinkV2Assertions.assertThat(toCommittableSummary(output.get(0)))
+        ListAssert<CommittableMessage<String>> records =
+                assertThat(testHarness.extractOutputValues()).hasSize(3);
+        records.element(0, as(committableSummary()))
                 .hasFailedCommittables(0)
                 .hasOverallCommittables(2)
                 .hasPendingCommittables(0);
-        SinkV2Assertions.assertThat(toCommittableWithLinage(output.get(1)))
-                .isEqualTo(first.withSubtaskId(0));
-        SinkV2Assertions.assertThat(toCommittableWithLinage(output.get(2)))
-                .isEqualTo(second.withSubtaskId(0));
+        records.element(1, as(committableWithLineage())).isEqualTo(first.withSubtaskId(0));
+        records.element(2, as(committableWithLineage())).isEqualTo(second.withSubtaskId(0));
         testHarness.close();
     }
 
@@ -226,24 +221,25 @@ abstract class CommitterOperatorTestBase {
         restored.open();
 
         // Previous committables are immediately committed if possible
-        final List<StreamElement> output = fromOutput(restored.getOutput());
-        assertThat(output).hasSize(3);
         assertThat(sinkAndCounters.commitCounter.getAsInt()).isEqualTo(2);
-        SinkV2Assertions.assertThat(toCommittableSummary(output.get(0)))
+        ListAssert<CommittableMessage<String>> records =
+                assertThat(restored.extractOutputValues()).hasSize(3);
+        records.element(0, as(committableSummary()))
                 .hasCheckpointId(checkpointId)
+                .hasSubtaskId(subtaskIdAfterRecovery)
                 .hasFailedCommittables(0)
                 .hasOverallCommittables(2)
                 .hasPendingCommittables(0);
 
         // Expect the same checkpointId that the original snapshot was made with.
-        SinkV2Assertions.assertThat(toCommittableWithLinage(output.get(1)))
-                .isEqualTo(
-                        new CommittableWithLineage<>(
-                                first.getCommittable(), checkpointId, subtaskIdAfterRecovery));
-        SinkV2Assertions.assertThat(toCommittableWithLinage(output.get(2)))
-                .isEqualTo(
-                        new CommittableWithLineage<>(
-                                second.getCommittable(), checkpointId, subtaskIdAfterRecovery));
+        records.element(1, as(committableWithLineage()))
+                .hasCheckpointId(checkpointId)
+                .hasSubtaskId(subtaskIdAfterRecovery)
+                .hasCommittable(first.getCommittable());
+        records.element(2, as(committableWithLineage()))
+                .hasCheckpointId(checkpointId)
+                .hasSubtaskId(subtaskIdAfterRecovery)
+                .hasCommittable(second.getCommittable());
         restored.close();
     }
 
@@ -275,14 +271,14 @@ abstract class CommitterOperatorTestBase {
             testHarness.notifyOfCompletedCheckpoint(1);
         }
 
-        final List<StreamElement> output = fromOutput(testHarness.getOutput());
-        assertThat(output).hasSize(2);
-        SinkV2Assertions.assertThat(toCommittableSummary(output.get(0)))
+        ListAssert<CommittableMessage<String>> records =
+                assertThat(testHarness.extractOutputValues()).hasSize(2);
+        records.element(0, as(committableSummary()))
                 .hasCheckpointId(1L)
                 .hasPendingCommittables(0)
                 .hasOverallCommittables(1)
                 .hasFailedCommittables(0);
-        SinkV2Assertions.assertThat(toCommittableWithLinage(output.get(1)))
+        records.element(1, as(committableWithLineage()))
                 .isEqualTo(committableWithLineage.withSubtaskId(0));
 
         // Future emission calls should change the output
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/SinkTestUtil.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/SinkTestUtil.java
index cfdae58596c..7d476fcc635 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/SinkTestUtil.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/SinkTestUtil.java
@@ -17,77 +17,13 @@
 
 package org.apache.flink.streaming.runtime.operators.sink;
 
-import org.apache.flink.core.io.SimpleVersionedSerialization;
-import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
 import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
 import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
 import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
-import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
-
-import java.io.IOException;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.List;
-import java.util.stream.Collectors;
 
 import static org.assertj.core.api.Assertions.assertThat;
 
 class SinkTestUtil {
-    static StreamRecord<byte[]> committableRecord(String element) {
-        return new StreamRecord<>(toBytes(element));
-    }
-
-    static List<StreamRecord<byte[]>> committableRecords(Collection<String> elements) {
-        return elements.stream().map(SinkTestUtil::committableRecord).collect(Collectors.toList());
-    }
-
-    static List<byte[]> toBytes(String... elements) {
-        return Arrays.stream(elements).map(SinkTestUtil::toBytes).collect(Collectors.toList());
-    }
-
-    static List<byte[]> toBytes(Collection<String> elements) {
-        return elements.stream().map(SinkTestUtil::toBytes).collect(Collectors.toList());
-    }
-
-    static byte[] toBytes(String obj) {
-        try {
-            return SimpleVersionedSerialization.writeVersionAndSerialize(
-                    TestSinkV2.COMMITTABLE_SERIALIZER, obj);
-        } catch (IOException e) {
-            throw new IllegalStateException(e);
-        }
-    }
-
-    static List<String> fromRecords(Collection<StreamRecord<byte[]>> elements) {
-        return elements.stream().map(SinkTestUtil::fromRecord).collect(Collectors.toList());
-    }
-
-    @SuppressWarnings("unchecked")
-    static List<StreamElement> fromOutput(Collection<Object> elements) {
-        return elements.stream()
-                .map(
-                        element -> {
-                            if (element instanceof StreamRecord) {
-                                return new StreamRecord<>(
-                                        ((StreamRecord<CommittableMessage<?>>) element).getValue());
-                            }
-                            return (StreamElement) element;
-                        })
-                .collect(Collectors.toList());
-    }
-
-    static String fromRecord(StreamRecord<byte[]> obj) {
-        return fromBytes(obj.getValue());
-    }
-
-    static String fromBytes(byte[] obj) {
-        try {
-            return SimpleVersionedSerialization.readVersionAndDeSerialize(
-                    TestSinkV2.COMMITTABLE_SERIALIZER, obj);
-        } catch (IOException e) {
-            throw new IllegalStateException(e);
-        }
-    }
 
     public static CommittableSummary<?> toCommittableSummary(StreamElement element) {
         final Object value = element.asRecord().getValue();
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/SinkWriterOperatorTestBase.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/SinkWriterOperatorTestBase.java
index d6a40dc24b3..46914441c13 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/SinkWriterOperatorTestBase.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/SinkWriterOperatorTestBase.java
@@ -36,19 +36,19 @@ import org.apache.flink.runtime.operators.testutils.MockEnvironment;
 import org.apache.flink.runtime.state.StateInitializationContext;
 import org.apache.flink.runtime.state.StateSnapshotContext;
 import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
-import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
 import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineageAssert;
 import org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions;
 import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
 import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
 import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
 import org.apache.flink.streaming.api.watermark.Watermark;
 import org.apache.flink.streaming.runtime.operators.sink.committables.SinkV1CommittableDeserializer;
-import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
 import org.apache.flink.streaming.util.TestHarnessUtil;
 
+import org.assertj.core.api.ListAssert;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.params.ParameterizedTest;
 import org.junit.jupiter.params.provider.ValueSource;
@@ -57,17 +57,19 @@ import java.io.IOException;
 import java.lang.reflect.Proxy;
 import java.util.ArrayList;
 import java.util.Arrays;
-import java.util.Collection;
 import java.util.Collections;
 import java.util.List;
 import java.util.Optional;
-import java.util.Queue;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.function.Consumer;
+import java.util.stream.Collectors;
 
+import static org.apache.flink.api.connector.sink2.InitContext.INITIAL_CHECKPOINT_ID;
 import static org.apache.flink.streaming.api.connector.sink2.CommittableMessage.EOI;
-import static org.apache.flink.streaming.runtime.operators.sink.SinkTestUtil.fromOutput;
+import static org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions.committableSummary;
+import static org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions.committableWithLineage;
+import static org.assertj.core.api.Assertions.as;
 import static org.assertj.core.api.Assertions.assertThat;
 
 abstract class SinkWriterOperatorTestBase {
@@ -81,12 +83,12 @@ abstract class SinkWriterOperatorTestBase {
         testHarness.open();
         testHarness.processElement(1, 1);
 
-        assertThat(testHarness.getOutput()).isEmpty();
+        assertThat(testHarness.extractOutputValues()).isEmpty();
         assertThat(sink.getRecordsOfCurrentCheckpoint())
                 .containsOnly("(1,1," + Long.MIN_VALUE + ")");
 
         testHarness.prepareSnapshotPreBarrier(1);
-        assertThat(testHarness.getOutput()).isEmpty();
+        assertThat(testHarness.extractOutputValues()).isEmpty();
         // Elements are flushed
         assertThat(sink.getRecordsOfCurrentCheckpoint()).isEmpty();
         testHarness.close();
@@ -132,14 +134,16 @@ abstract class SinkWriterOperatorTestBase {
         testHarness.prepareSnapshotPreBarrier(1L);
 
         // Expect empty committableSummary
-        assertBasicOutput(testHarness.getOutput(), 0, 1L);
-        testHarness.getOutput().poll();
+        assertBasicOutput(testHarness.extractOutputValues(), 0, 1L);
 
         testHarness.getProcessingTimeService().setCurrentTime(2001);
 
         testHarness.prepareSnapshotPreBarrier(2L);
 
-        assertBasicOutput(testHarness.getOutput(), 2, 2L);
+        assertBasicOutput(
+                testHarness.extractOutputValues().stream().skip(1).collect(Collectors.toList()),
+                2,
+                2L);
         testHarness.close();
     }
 
@@ -150,7 +154,7 @@ abstract class SinkWriterOperatorTestBase {
                         new SinkWriterOperatorFactory<>(sinkWithCommitter().getSink()));
 
         testHarness.open();
-        assertThat(testHarness.getOutput()).isEmpty();
+        assertThat(testHarness.extractOutputValues()).isEmpty();
 
         testHarness.processElement(1, 1);
         testHarness.processElement(2, 2);
@@ -158,7 +162,7 @@ abstract class SinkWriterOperatorTestBase {
         // flush
         testHarness.prepareSnapshotPreBarrier(1);
 
-        assertBasicOutput(testHarness.getOutput(), 2, 1L);
+        assertBasicOutput(testHarness.extractOutputValues(), 2, 1L);
         testHarness.close();
     }
 
@@ -170,11 +174,11 @@ abstract class SinkWriterOperatorTestBase {
                 new OneInputStreamOperatorTestHarness<>(writerOperatorFactory);
 
         testHarness.open();
-        assertThat(testHarness.getOutput()).isEmpty();
+        assertThat(testHarness.extractOutputValues()).isEmpty();
 
         testHarness.processElement(1, 1);
         testHarness.endInput();
-        assertBasicOutput(testHarness.getOutput(), 1, EOI);
+        assertBasicOutput(testHarness.extractOutputValues(), 1, EOI);
     }
 
     @ParameterizedTest
@@ -292,46 +296,40 @@ abstract class SinkWriterOperatorTestBase {
 
         testHarness.prepareSnapshotPreBarrier(2);
 
-        final List<StreamElement> output = fromOutput(testHarness.getOutput());
-        assertThat(output).hasSize(4);
-
-        assertThat(output.get(0).asRecord().getValue())
-                .isInstanceOf(CommittableSummary.class)
-                .satisfies(
-                        cs ->
-                                SinkV2Assertions.assertThat(((CommittableSummary<?>) cs))
-                                        .hasPendingCommittables(committables.size())
-                                        .hasCheckpointId(
-                                                org.apache.flink.api.connector.sink2.Sink
-                                                        .InitContext.INITIAL_CHECKPOINT_ID)
-                                        .hasOverallCommittables(committables.size())
-                                        .hasFailedCommittables(0));
-        assertRestoredCommitterCommittable(
-                output.get(1).asRecord().getValue(), committables.get(0));
-        assertRestoredCommitterCommittable(
-                output.get(2).asRecord().getValue(), committables.get(1));
-        assertThat(output.get(3).asRecord().getValue())
-                .isInstanceOf(CommittableSummary.class)
-                .satisfies(
-                        cs ->
-                                SinkV2Assertions.assertThat(((CommittableSummary<?>) cs))
-                                        .hasPendingCommittables(0)
-                                        .hasCheckpointId(2L)
-                                        .hasOverallCommittables(0)
-                                        .hasFailedCommittables(0));
+        final ListAssert<CommittableMessage<Integer>> records =
+                assertThat(testHarness.extractOutputValues()).hasSize(4);
+
+        records.element(0, as(committableSummary()))
+                .hasPendingCommittables(committables.size())
+                .hasCheckpointId(INITIAL_CHECKPOINT_ID)
+                .hasOverallCommittables(committables.size())
+                .hasFailedCommittables(0);
+        records.<CommittableWithLineageAssert<String>>element(1, as(committableWithLineage()))
+                .hasCommittable(committables.get(0))
+                .hasCheckpointId(INITIAL_CHECKPOINT_ID)
+                .hasSubtaskId(0);
+        records.<CommittableWithLineageAssert<String>>element(2, as(committableWithLineage()))
+                .hasCommittable(committables.get(1))
+                .hasCheckpointId(INITIAL_CHECKPOINT_ID)
+                .hasSubtaskId(0);
+        records.element(3, as(committableSummary()))
+                .hasPendingCommittables(0)
+                .hasCheckpointId(2L)
+                .hasOverallCommittables(0)
+                .hasFailedCommittables(0);
     }
 
     @ParameterizedTest
     @ValueSource(booleans = {true, false})
     void testHandleEndInputInStreamingMode(boolean isCheckpointingEnabled) throws Exception {
         InspectableSink sink = sinkWithCommitter();
-        final OneInputStreamOperatorTestHarness<Integer, CommittableMessage<Integer>> testHarness =
+        final OneInputStreamOperatorTestHarness<Integer, CommittableMessage<String>> testHarness =
                 new OneInputStreamOperatorTestHarness<>(
                         new SinkWriterOperatorFactory<>(sink.getSink()));
         testHarness.open();
         testHarness.processElement(1, 1);
 
-        assertThat(testHarness.getOutput()).isEmpty();
+        assertThat(testHarness.extractOutputValues()).isEmpty();
         final String record = "(1,1," + Long.MIN_VALUE + ")";
         assertThat(sink.getRecordsOfCurrentCheckpoint()).containsOnly(record);
 
@@ -341,7 +339,18 @@ abstract class SinkWriterOperatorTestBase {
             testHarness.prepareSnapshotPreBarrier(1);
         }
 
-        assertEmitted(Collections.singletonList(record), testHarness.getOutput());
+        List<String> committables = Collections.singletonList(record);
+
+        ListAssert<CommittableMessage<String>> records =
+                assertThat(testHarness.extractOutputValues()).hasSize(committables.size() + 1);
+        records.element(0, as(committableSummary()))
+                .hasPendingCommittables(committables.size())
+                .hasOverallCommittables(committables.size())
+                .hasFailedCommittables(0);
+
+        records.filteredOn(message -> message instanceof CommittableWithLineage)
+                .map(message -> ((CommittableWithLineage<String>) message).getCommittable())
+                .containsExactlyInAnyOrderElementsOf(committables);
         assertThat(sink.getRecordsOfCurrentCheckpoint()).isEmpty();
 
         testHarness.close();
@@ -480,67 +489,22 @@ abstract class SinkWriterOperatorTestBase {
         assertThat(initContext.metadataConsumer()).isEqualTo(original.metadataConsumer());
     }
 
-    @SuppressWarnings("unchecked")
-    private static void assertRestoredCommitterCommittable(Object record, String committable) {
-        assertThat(record)
-                .isInstanceOf(CommittableWithLineage.class)
-                .satisfies(
+    private static void assertBasicOutput(
+            List<CommittableMessage<Integer>> output, int numberOfCommittables, long checkpointId) {
+        ListAssert<CommittableMessage<Integer>> records =
+                assertThat(output).hasSize(numberOfCommittables + 1);
+        records.element(0, as(committableSummary()))
+                .hasOverallCommittables(numberOfCommittables)
+                .hasPendingCommittables(numberOfCommittables)
+                .hasFailedCommittables(0);
+        records.filteredOn(r -> r instanceof CommittableWithLineage)
+                .allSatisfy(
                         cl ->
-                                SinkV2Assertions.assertThat((CommittableWithLineage<String>) cl)
-                                        .hasCommittable(committable)
-                                        .hasCheckpointId(
-                                                org.apache.flink.api.connector.sink2.Sink
-                                                        .InitContext.INITIAL_CHECKPOINT_ID)
+                                SinkV2Assertions.assertThat((CommittableWithLineage<?>) cl)
+                                        .hasCheckpointId(checkpointId)
                                         .hasSubtaskId(0));
     }
 
-    @SuppressWarnings("unchecked")
-    private static void assertEmitted(List<String> records, Queue<Object> output) {
-
-        final List<StreamElement> collected = fromOutput(output);
-        assertThat(collected).hasSize(records.size() + 1);
-        assertThat(collected.get(0).asRecord().getValue())
-                .isInstanceOf(CommittableSummary.class)
-                .satisfies(
-                        cs ->
-                                SinkV2Assertions.assertThat(((CommittableSummary<?>) cs))
-                                        .hasPendingCommittables(records.size())
-                                        .hasOverallCommittables(records.size())
-                                        .hasFailedCommittables(0));
-
-        final List<String> committables = new ArrayList<>();
-
-        for (int i = 1; i <= records.size(); i++) {
-            Object value = collected.get(i).asRecord().getValue();
-            assertThat(value).isInstanceOf(CommittableWithLineage.class);
-            committables.add(((CommittableWithLineage<String>) value).getCommittable());
-        }
-        assertThat(committables).containsExactlyInAnyOrderElementsOf(records);
-    }
-
-    private static void assertBasicOutput(
-            Collection<Object> queuedOutput, int numberOfCommittables, long checkpointId) {
-        List<StreamElement> output = fromOutput(queuedOutput);
-        assertThat(output).hasSize(numberOfCommittables + 1);
-        assertThat(output.get(0).asRecord().getValue())
-                .isInstanceOf(CommittableSummary.class)
-                .satisfies(
-                        cs ->
-                                SinkV2Assertions.assertThat((CommittableSummary<?>) cs)
-                                        .hasOverallCommittables(numberOfCommittables)
-                                        .hasPendingCommittables(numberOfCommittables)
-                                        .hasFailedCommittables(0));
-        for (int i = 1; i <= numberOfCommittables; i++) {
-            assertThat(output.get(i).asRecord().getValue())
-                    .isInstanceOf(CommittableWithLineage.class)
-                    .satisfies(
-                            cl ->
-                                    SinkV2Assertions.assertThat((CommittableWithLineage<?>) cl)
-                                            .hasCheckpointId(checkpointId)
-                                            .hasSubtaskId(0));
-        }
-    }
-
     private static class TestCommitterOperator extends AbstractStreamOperator<String>
             implements OneInputStreamOperator<String, String> {
 
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollectorSerializerTest.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollectorSerializerTest.java
index a4125b359d4..7c9dbb81779 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollectorSerializerTest.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollectorSerializerTest.java
@@ -29,9 +29,6 @@ import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
 import org.apache.flink.streaming.api.connector.sink2.IntegerSerializer;
 import org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions;
 
-import org.apache.flink.shaded.guava31.com.google.common.collect.Streams;
-
-import org.apache.commons.lang3.tuple.Pair;
 import org.assertj.core.api.InstanceOfAssertFactories;
 import org.junit.jupiter.api.Test;
 
@@ -43,7 +40,6 @@ import java.util.List;
 import java.util.stream.Collectors;
 
 import static org.assertj.core.api.Assertions.assertThat;
-import static org.junit.jupiter.api.Assertions.assertAll;
 
 class CommittableCollectorSerializerTest {
 
@@ -200,45 +196,27 @@ class CommittableCollectorSerializerTest {
             CommittableCollector<Integer> committableCollector,
             List<List<Integer>> committablesPerSubtaskPerCheckpoint) {
 
-        assertAll(
-                assertMessageHeading,
-                () -> {
-                    final Collection<CheckpointCommittableManagerImpl<Integer>>
-                            checkpointCommittables =
-                                    committableCollector.getCheckpointCommittables();
-                    final int expectedCommittableSize = committablesPerSubtaskPerCheckpoint.size();
-                    assertThat(checkpointCommittables).hasSize(expectedCommittableSize);
-
-                    Streams.zip(
-                                    checkpointCommittables.stream(),
-                                    committablesPerSubtaskPerCheckpoint.stream(),
-                                    Pair::of)
-                            .forEach(
-                                    pair -> {
-                                        CheckpointCommittableManagerImpl<Integer>
-                                                checkpointCommittableManager = pair.getKey();
-                                        List<Integer> expectedPendingRequestCount = pair.getValue();
-
-                                        final SubtaskCommittableManager<Integer>
-                                                subtaskCommittableManager =
-                                                        checkpointCommittableManager
-                                                                .getSubtaskCommittableManager(
-                                                                        subtaskId);
-
-                                        SinkV2Assertions.assertThat(
-                                                        checkpointCommittableManager.getSummary(
-                                                                subtaskId, numberOfSubtasks))
-                                                .hasSubtaskId(subtaskId)
-                                                .hasNumberOfSubtasks(numberOfSubtasks);
-
-                                        assertPendingRequests(
-                                                subtaskCommittableManager,
-                                                expectedPendingRequestCount);
-
-                                        assertThat(subtaskCommittableManager.getSubtaskId())
-                                                .isEqualTo(subtaskId);
-                                    });
-                });
+        assertThat(committableCollector.getCheckpointCommittables())
+                .describedAs(assertMessageHeading)
+                .zipSatisfy(
+                        committablesPerSubtaskPerCheckpoint,
+                        (checkpointCommittableManager, expectedPendingRequestCount) -> {
+                            final SubtaskCommittableManager<Integer> subtaskCommittableManager =
+                                    checkpointCommittableManager.getSubtaskCommittableManager(
+                                            subtaskId);
+
+                            SinkV2Assertions.assertThat(
+                                            checkpointCommittableManager.getSummary(
+                                                    subtaskId, numberOfSubtasks))
+                                    .hasSubtaskId(subtaskId)
+                                    .hasNumberOfSubtasks(numberOfSubtasks);
+
+                            assertPendingRequests(
+                                    subtaskCommittableManager, expectedPendingRequestCount);
+
+                            assertThat(subtaskCommittableManager.getSubtaskId())
+                                    .isEqualTo(subtaskId);
+                        });
     }
 
     private void assertPendingRequests(
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittableManagerTest.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittableManagerTest.java
index 32f98a2b623..78d884c17cd 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittableManagerTest.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/committables/SubtaskCommittableManagerTest.java
@@ -22,15 +22,16 @@ import org.apache.flink.metrics.groups.SinkCommitterMetricGroup;
 import org.apache.flink.runtime.metrics.groups.MetricsGroupTestUtils;
 import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
 
+import org.assertj.core.api.ListAssert;
 import org.junit.jupiter.api.Test;
 
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Iterator;
-import java.util.List;
 import java.util.stream.IntStream;
 
-import static org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions.assertThat;
+import static org.apache.flink.streaming.api.connector.sink2.SinkV2Assertions.committableWithLineage;
+import static org.assertj.core.api.Assertions.as;
 import static org.assertj.core.api.Assertions.assertThat;
 
 class SubtaskCommittableManagerTest {
@@ -65,11 +66,18 @@ class SubtaskCommittableManagerTest {
         assertThat(subtaskCommittableManager.getNumDrained()).isEqualTo(0);
 
         // Drain committed committables
-        final List<CommittableWithLineage<Integer>> committables =
-                subtaskCommittableManager.drainCommitted();
-        assertThat(committables).hasSize(2);
-        assertThat(committables.get(0)).hasSubtaskId(1).hasCommittable(1).hasCheckpointId(1);
-        assertThat(committables.get(1)).hasSubtaskId(1).hasCommittable(2).hasCheckpointId(1);
+        ListAssert<CommittableWithLineage<Integer>> committables =
+                assertThat(subtaskCommittableManager.drainCommitted()).hasSize(2);
+        committables
+                .element(0, as(committableWithLineage()))
+                .hasSubtaskId(1)
+                .hasCommittable(1)
+                .hasCheckpointId(1);
+        committables
+                .element(1, as(committableWithLineage()))
+                .hasSubtaskId(1)
+                .hasCommittable(2)
+                .hasCheckpointId(1);
         assertThat(subtaskCommittableManager.getNumFailed()).isEqualTo(0);
 
         // Drain again should not yield anything


(flink) 04/04: [FLINK-36788] Fix GlobalCommitter expansion

Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

arvid pushed a commit to branch release-1.20
in repository https://gitbox.apache.org/repos/asf/flink.git

commit 504c866b347764b242d3605893533923587ef9ab
Author: Arvid Heise <ar...@apache.org>
AuthorDate: Fri Nov 29 15:08:13 2024 +0100

    [FLINK-36788] Fix GlobalCommitter expansion
    
    So far, the global committer didn't properly get all properties set during expansion. In some cases, it didn't get expanded at all.
    
    This commit also inline SinkTransformationTranslatorITCaseBase.java after the respective V1 implementation was removed. V2 now has the same coverage as V1 used to have.
---
 .../GlobalCommitterTransformationTranslator.java   | 23 ++++++--
 .../translators/SinkTransformationTranslator.java  | 18 +++++-
 .../SinkTransformationTranslatorITCaseBase.java    | 63 +++++++++++++++++++++
 .../SinkV1TransformationTranslatorITCase.java      | 64 +++++-----------------
 ...V2TransformationTranslatorDeprecatedITCase.java | 14 +++++
 .../SinkV2TransformationTranslatorITCase.java      | 14 +++++
 .../runtime/operators/sink/TestSinkV2.java         | 15 ++---
 .../operators/sink/deprecated/TestSinkV2.java      |  4 +-
 8 files changed, 147 insertions(+), 68 deletions(-)

diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/GlobalCommitterTransformationTranslator.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/GlobalCommitterTransformationTranslator.java
index 69c4533b362..6147e8799d9 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/GlobalCommitterTransformationTranslator.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/GlobalCommitterTransformationTranslator.java
@@ -34,11 +34,14 @@ import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactor
 import org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperatorFactory;
 
 import java.util.ArrayDeque;
+import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
 import java.util.HashSet;
 import java.util.Queue;
 import java.util.Set;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
 
 import static org.apache.flink.streaming.api.connector.sink2.StandardSinkTopologies.GLOBAL_COMMITTER_TRANSFORMATION_NAME;
 
@@ -74,11 +77,10 @@ public class GlobalCommitterTransformationTranslator<CommT>
         boolean commitOnInput = batch || !checkpointingEnabled || hasUpstreamCommitter(inputStream);
 
         // Create a global shuffle and add the global committer with parallelism 1.
+        DataStream<CommittableMessage<CommT>> global = inputStream.global();
         final PhysicalTransformation<Void> transformation =
                 (PhysicalTransformation<Void>)
-                        inputStream
-                                .global()
-                                .transform(
+                        global.transform(
                                         GLOBAL_COMMITTER_TRANSFORMATION_NAME,
                                         Types.VOID,
                                         new GlobalCommitterOperator<>(
@@ -87,10 +89,20 @@ public class GlobalCommitterTransformationTranslator<CommT>
                                                 commitOnInput))
                                 .getTransformation();
         transformation.setChainingStrategy(ChainingStrategy.ALWAYS);
-        transformation.setName(GLOBAL_COMMITTER_TRANSFORMATION_NAME);
         transformation.setParallelism(1);
         transformation.setMaxParallelism(1);
-        return Collections.emptyList();
+        copySafely(transformation::setName, globalCommitterTransform::getName);
+        copySafely(transformation::setUid, globalCommitterTransform::getUid);
+        copySafely(transformation::setUidHash, globalCommitterTransform::getUserProvidedNodeHash);
+
+        return Arrays.asList(global.getId(), transformation.getId());
+    }
+
+    private static <T> void copySafely(Consumer<T> consumer, Supplier<T> provider) {
+        T value = provider.get();
+        if (value != null) {
+            consumer.accept(value);
+        }
     }
 
     /**
@@ -109,7 +121,6 @@ public class GlobalCommitterTransformationTranslator<CommT>
                 StreamOperatorFactory<?> operatorFactory =
                         ((OneInputTransformation<?, ?>) transformation).getOperatorFactory();
                 if (operatorFactory instanceof CommitterOperatorFactory) {
-                    // found the committer, so the global committer should commit on input
                     return true;
                 }
                 if (operatorFactory instanceof SinkWriterOperatorFactory) {
diff --git a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
index 24cbe2d2e81..dd41218a497 100644
--- a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
+++ b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SinkTransformationTranslator.java
@@ -59,12 +59,14 @@ import java.util.Queue;
 import java.util.Set;
 import java.util.function.BiConsumer;
 import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
 
 import static org.apache.flink.util.Preconditions.checkState;
 
 /**
  * A {@link org.apache.flink.streaming.api.graph.TransformationTranslator} for the {@link
- * org.apache.flink.streaming.api.transformations.SinkTransformation}.
+ * SinkTransformation}.
  */
 @Internal
 public class SinkTransformationTranslator<Input, Output>
@@ -177,6 +179,12 @@ public class SinkTransformationTranslator<Input, Output>
 
             getSinkTransformations(sizeBefore).forEach(context::transform);
 
+            repeatUntilConverged(
+                    () ->
+                            getSinkTransformations(sizeBefore).stream()
+                                    .flatMap(t -> context.transform(t).stream())
+                                    .collect(Collectors.toList()));
+
             disallowUnalignedCheckpoint(getSinkTransformations(sizeBefore));
 
             // Remove all added sink subtransformations to avoid duplications and allow additional
@@ -188,6 +196,14 @@ public class SinkTransformationTranslator<Input, Output>
             }
         }
 
+        private <R> void repeatUntilConverged(Supplier<R> producer) {
+            R lastResult = producer.get();
+            R nextResult;
+            while (!lastResult.equals(nextResult = producer.get())) {
+                lastResult = nextResult;
+            }
+        }
+
         private List<Transformation<?>> getSinkTransformations(int sizeBefore) {
             return executionEnvironment
                     .getTransformations()
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkTransformationTranslatorITCaseBase.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkTransformationTranslatorITCaseBase.java
index e95523a1492..5f976e2684c 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkTransformationTranslatorITCaseBase.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkTransformationTranslatorITCaseBase.java
@@ -23,6 +23,7 @@ import org.apache.flink.api.common.typeutils.base.IntSerializer;
 import org.apache.flink.configuration.Configuration;
 import org.apache.flink.configuration.ExecutionOptions;
 import org.apache.flink.core.io.SimpleVersionedSerializerTypeSerializerProxy;
+import org.apache.flink.streaming.api.datastream.CustomSinkOperatorUidHashes;
 import org.apache.flink.streaming.api.datastream.DataStream;
 import org.apache.flink.streaming.api.datastream.DataStreamSink;
 import org.apache.flink.streaming.api.datastream.DataStreamSource;
@@ -35,6 +36,7 @@ import org.apache.flink.testutils.junit.extensions.parameterized.Parameter;
 import org.apache.flink.testutils.junit.extensions.parameterized.ParameterizedTestExtension;
 import org.apache.flink.testutils.junit.extensions.parameterized.Parameters;
 
+import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.TestTemplate;
 import org.junit.jupiter.api.extension.ExtendWith;
 
@@ -69,8 +71,13 @@ abstract class SinkTransformationTranslatorITCaseBase<SinkT> {
 
     abstract SinkT sinkWithCommitter();
 
+    abstract SinkT sinkWithGlobalCommitter();
+
     abstract DataStreamSink<Integer> sinkTo(DataStream<Integer> stream, SinkT sink);
 
+    abstract DataStreamSink<Integer> sinkTo(
+            DataStream<Integer> stream, SinkT sink, CustomSinkOperatorUidHashes hashes);
+
     @TestTemplate
     void generateWriterTopology() {
         final StreamGraph streamGraph = buildGraph(simpleSink(), runtimeExecutionMode);
@@ -127,6 +134,62 @@ abstract class SinkTransformationTranslatorITCaseBase<SinkT> {
         testParallelismConfiguredInternal(false);
     }
 
+    @TestTemplate
+    void testSettingOperatorUidHash() {
+        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+        final DataStreamSource<Integer> src = env.fromData(1, 2);
+        final String writerHash = "f6b178ce445dc3ffaa06bad27a51fead";
+        final String committerHash = "68ac8ae79eae4e3135a54f9689c4aa10";
+        final String globalCommitterHash = "77e6aa6eeb1643b3765e1e4a7a672f37";
+        final CustomSinkOperatorUidHashes operatorsUidHashes =
+                CustomSinkOperatorUidHashes.builder()
+                        .setWriterUidHash(writerHash)
+                        .setCommitterUidHash(committerHash)
+                        .setGlobalCommitterUidHash(globalCommitterHash)
+                        .build();
+        sinkTo(src, sinkWithGlobalCommitter(), operatorsUidHashes).name(NAME);
+
+        final StreamGraph streamGraph = env.getStreamGraph();
+
+        assertThat(findWriter(streamGraph).getUserHash()).isEqualTo(writerHash);
+        assertThat(findCommitter(streamGraph).getUserHash()).isEqualTo(committerHash);
+        assertThat(findGlobalCommitter(streamGraph).getUserHash()).isEqualTo(globalCommitterHash);
+    }
+
+    /**
+     * When ever you need to change something in this test case please think about possible state
+     * upgrade problems introduced by your changes.
+     */
+    @TestTemplate
+    void testSettingOperatorUids() {
+        final String sinkUid = "f6b178ce445dc3ffaa06bad27a51fead";
+        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+        final DataStreamSource<Integer> src = env.fromData(1, 2);
+        sinkTo(src, sinkWithGlobalCommitter()).name(NAME).uid(sinkUid);
+
+        final StreamGraph streamGraph = env.getStreamGraph();
+        assertThat(findWriter(streamGraph).getTransformationUID()).isEqualTo(sinkUid);
+        assertThat(findCommitter(streamGraph).getTransformationUID())
+                .isEqualTo(String.format("Sink Committer: %s", sinkUid));
+        assertThat(findGlobalCommitter(streamGraph).getTransformationUID())
+                .isEqualTo(String.format("Sink %s Global Committer", sinkUid));
+    }
+
+    @Test
+    void testSettingOperatorNames() {
+        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+        final DataStreamSource<Integer> src = env.fromData(1, 2);
+        sinkTo(src, sinkWithGlobalCommitter()).name(NAME);
+
+        final StreamGraph streamGraph = env.getStreamGraph();
+        assertThat(findWriter(streamGraph).getOperatorName())
+                .isEqualTo(String.format("%s: Writer", NAME));
+        assertThat(findCommitter(streamGraph).getOperatorName())
+                .isEqualTo(String.format("%s: Committer", NAME));
+        assertThat(findGlobalCommitter(streamGraph).getOperatorName())
+                .isEqualTo(String.format("%s: Global Committer", NAME));
+    }
+
     private void testParallelismConfiguredInternal(boolean setSinkParallelism) {
         final StreamGraph streamGraph =
                 buildGraph(sinkWithCommitter(), runtimeExecutionMode, setSinkParallelism);
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV1TransformationTranslatorITCase.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV1TransformationTranslatorITCase.java
index d9003e2b3a8..7d00a736c11 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV1TransformationTranslatorITCase.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV1TransformationTranslatorITCase.java
@@ -25,8 +25,6 @@ import org.apache.flink.core.io.SimpleVersionedSerializerTypeSerializerProxy;
 import org.apache.flink.streaming.api.datastream.CustomSinkOperatorUidHashes;
 import org.apache.flink.streaming.api.datastream.DataStream;
 import org.apache.flink.streaming.api.datastream.DataStreamSink;
-import org.apache.flink.streaming.api.datastream.DataStreamSource;
-import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 import org.apache.flink.streaming.api.operators.SimpleOperatorFactory;
 import org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory;
 import org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperatorFactory;
@@ -57,11 +55,24 @@ class SinkV1TransformationTranslatorITCase
         return TestSink.newBuilder().setDefaultCommitter().build();
     }
 
+    @Override
+    Sink<Integer, ?, ?, ?> sinkWithGlobalCommitter() {
+        return TestSink.newBuilder().setDefaultCommitter().setDefaultGlobalCommitter().build();
+    }
+
     @Override
     DataStreamSink<Integer> sinkTo(DataStream<Integer> stream, Sink<Integer, ?, ?, ?> sink) {
         return stream.sinkTo(sink);
     }
 
+    @Override
+    DataStreamSink<Integer> sinkTo(
+            DataStream<Integer> stream,
+            Sink<Integer, ?, ?, ?> sink,
+            CustomSinkOperatorUidHashes hashes) {
+        return stream.sinkTo(sink, hashes);
+    }
+
     @TestTemplate
     void generateWriterCommitterGlobalCommitterTopology() {
 
@@ -156,53 +167,4 @@ class SinkV1TransformationTranslatorITCase
                 1,
                 1);
     }
-
-    @TestTemplate
-    void testSettingOperatorUidHash() {
-        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-        final DataStreamSource<Integer> src = env.fromData(1, 2);
-        final String writerHash = "f6b178ce445dc3ffaa06bad27a51fead";
-        final String committerHash = "68ac8ae79eae4e3135a54f9689c4aa10";
-        final String globalCommitterHash = "77e6aa6eeb1643b3765e1e4a7a672f37";
-        final CustomSinkOperatorUidHashes operatorsUidHashes =
-                CustomSinkOperatorUidHashes.builder()
-                        .setWriterUidHash(writerHash)
-                        .setCommitterUidHash(committerHash)
-                        .setGlobalCommitterUidHash(globalCommitterHash)
-                        .build();
-        src.sinkTo(
-                        TestSink.newBuilder()
-                                .setDefaultCommitter()
-                                .setDefaultGlobalCommitter()
-                                .build(),
-                        operatorsUidHashes)
-                .name(NAME);
-
-        final StreamGraph streamGraph = env.getStreamGraph();
-
-        assertThat(findWriter(streamGraph).getUserHash()).isEqualTo(writerHash);
-        assertThat(findCommitter(streamGraph).getUserHash()).isEqualTo(committerHash);
-        assertThat(findGlobalCommitter(streamGraph).getUserHash()).isEqualTo(globalCommitterHash);
-    }
-
-    /**
-     * When ever you need to change something in this test case please think about possible state
-     * upgrade problems introduced by your changes.
-     */
-    @TestTemplate
-    void testSettingOperatorUids() {
-        final String sinkUid = "f6b178ce445dc3ffaa06bad27a51fead";
-        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-        final DataStreamSource<Integer> src = env.fromData(1, 2);
-        src.sinkTo(TestSink.newBuilder().setDefaultCommitter().setDefaultGlobalCommitter().build())
-                .name(NAME)
-                .uid(sinkUid);
-
-        final StreamGraph streamGraph = env.getStreamGraph();
-        assertThat(findWriter(streamGraph).getTransformationUID()).isEqualTo(sinkUid);
-        assertThat(findCommitter(streamGraph).getTransformationUID())
-                .isEqualTo(String.format("Sink Committer: %s", sinkUid));
-        assertThat(findGlobalCommitter(streamGraph).getTransformationUID())
-                .isEqualTo(String.format("Sink %s Global Committer", sinkUid));
-    }
 }
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV2TransformationTranslatorDeprecatedITCase.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV2TransformationTranslatorDeprecatedITCase.java
index 20cd837dc27..2d6256393db 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV2TransformationTranslatorDeprecatedITCase.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV2TransformationTranslatorDeprecatedITCase.java
@@ -55,11 +55,25 @@ class SinkV2TransformationTranslatorDeprecatedITCase
         return TestSinkV2.<Integer>newBuilder().setDefaultCommitter().build();
     }
 
+    @Override
+    Sink<Integer> sinkWithGlobalCommitter() {
+        return TestSinkV2.<Integer>newBuilder()
+                .setDefaultCommitter()
+                .setWithPostCommitTopology(true)
+                .build();
+    }
+
     @Override
     DataStreamSink<Integer> sinkTo(DataStream<Integer> stream, Sink<Integer> sink) {
         return stream.sinkTo(sink);
     }
 
+    @Override
+    DataStreamSink<Integer> sinkTo(
+            DataStream<Integer> stream, Sink<Integer> sink, CustomSinkOperatorUidHashes hashes) {
+        return stream.sinkTo(sink, hashes);
+    }
+
     @TestTemplate
     void testSettingOperatorUidHash() {
         final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV2TransformationTranslatorITCase.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV2TransformationTranslatorITCase.java
index 6b09b83961d..16e09ed9046 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV2TransformationTranslatorITCase.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SinkV2TransformationTranslatorITCase.java
@@ -51,11 +51,25 @@ class SinkV2TransformationTranslatorITCase
         return TestSinkV2.<Integer>newBuilder().setDefaultCommitter().build();
     }
 
+    @Override
+    Sink<Integer> sinkWithGlobalCommitter() {
+        return TestSinkV2.<Integer>newBuilder()
+                .setDefaultCommitter()
+                .setWithPostCommitTopology(true)
+                .build();
+    }
+
     @Override
     DataStreamSink<Integer> sinkTo(DataStream<Integer> stream, Sink<Integer> sink) {
         return stream.sinkTo(sink);
     }
 
+    @Override
+    DataStreamSink<Integer> sinkTo(
+            DataStream<Integer> stream, Sink<Integer> sink, CustomSinkOperatorUidHashes hashes) {
+        return stream.sinkTo(sink, hashes);
+    }
+
     @TestTemplate
     void testSettingOperatorUidHash() {
         final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/TestSinkV2.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/TestSinkV2.java
index 7960478a784..0a34d157861 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/TestSinkV2.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/TestSinkV2.java
@@ -37,6 +37,7 @@ import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
 import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
 import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
 import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.connector.sink2.StandardSinkTopologies;
 import org.apache.flink.streaming.api.connector.sink2.SupportsPostCommitTopology;
 import org.apache.flink.streaming.api.connector.sink2.SupportsPreCommitTopology;
 import org.apache.flink.streaming.api.datastream.DataStream;
@@ -225,11 +226,6 @@ public class TestSinkV2<InputT> implements Sink<InputT> {
         public SimpleVersionedSerializer<String> getCommittableSerializer() {
             return committableSerializer;
         }
-
-        @Override
-        public SinkWriter<InputT> createWriter(WriterInitContext context) throws IOException {
-            return super.createWriter(context);
-        }
     }
 
     // -------------------------------------- Sink With PostCommitTopology -------------------------
@@ -246,7 +242,8 @@ public class TestSinkV2<InputT> implements Sink<InputT> {
 
         @Override
         public void addPostCommitTopology(DataStream<CommittableMessage<String>> committables) {
-            // We do not need to do anything for tests
+            StandardSinkTopologies.addGlobalCommitter(
+                    committables, DefaultCommitter::new, this::getCommittableSerializer);
         }
     }
 
@@ -468,17 +465,17 @@ public class TestSinkV2<InputT> implements Sink<InputT> {
     /** A {@link Committer} that always re-commits the committables data it received. */
     static class RetryOnceCommitter extends DefaultCommitter {
 
-        private final Set<CommitRequest<String>> seen = new LinkedHashSet<>();
+        private final Set<String> seen = new LinkedHashSet<>();
 
         @Override
         public void commit(Collection<CommitRequest<String>> committables) {
             committables.forEach(
                     c -> {
-                        if (seen.remove(c)) {
+                        if (seen.remove(c.getCommittable())) {
                             checkNotNull(committedData);
                             committedData.add(c);
                         } else {
-                            seen.add(c);
+                            seen.add(c.getCommittable());
                             c.retryLater();
                         }
                     });
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/deprecated/TestSinkV2.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/deprecated/TestSinkV2.java
index 217b57230c6..09e8be690b7 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/deprecated/TestSinkV2.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/sink/deprecated/TestSinkV2.java
@@ -28,6 +28,7 @@ import org.apache.flink.api.java.tuple.Tuple3;
 import org.apache.flink.core.io.SimpleVersionedSerializer;
 import org.apache.flink.core.io.SimpleVersionedSerializerAdapter;
 import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.StandardSinkTopologies;
 import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
 import org.apache.flink.streaming.api.datastream.DataStream;
 import org.apache.flink.util.Preconditions;
@@ -215,7 +216,8 @@ public class TestSinkV2<InputT> implements Sink<InputT> {
 
         @Override
         public void addPostCommitTopology(DataStream<CommittableMessage<String>> committables) {
-            // We do not need to do anything for tests
+            StandardSinkTopologies.addGlobalCommitter(
+                    committables, DefaultCommitter::new, this::getCommittableSerializer);
         }
     }